You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/01/20 16:16:22 UTC

[GitHub] [flink] fapaul opened a new pull request #18428: [Flink 25575] Add Sink V2 operators and translation

fapaul opened a new pull request #18428:
URL: https://github.com/apache/flink/pull/18428


   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   This PR allows executing Sink V2 and migrates the internal operator representation of Sink V1 to also use the Sink V2 operators. So basically every Sink V1 now runs internally on the Sink V2 infrastructure. Only the GlobalCommitter is currently still missing but it is tracked as a followup in https://issues.apache.org/jira/browse/FLINK-25726.
   
   ## Brief change log
   
   - d45e3a669eece2fc287d454ed4d69a5a34c9dff9 This adds the internal structure send and book keep the Committable Messages it ensures that the operators working with CommittableMessages can ensure exactly-once processing
   - 029d72ebef1e5df84313ea9c23e86ee0ccaecda7 small change to expose the held transformations of the stream environment. It is mainly used later by the SinkTransformationTranslator to ingest newly added transformations to the StreamGraphGenerator
   - 90556071ebceb2a01768bae0db671d2a808f8e23 Batch/Blocking exchanges are now changed by the framework accordingly in Streaming Mode. They only have an effect in Batch Mode.
   - 5ba4487b7ade1c3066ae695183d2723427aa8d32 Introduce interface to forward information about the current execution mode to operator factories
   - fa0eaa1541ce6d05fe8460c060c45cb0f6bb196d Adds a simulation adapter to convert Sink V1 to Sink V2.
   - f799eeea34b4d3c5691cc4b77f5be103ef1c065c Implements the operator logic and adapts the SinkTransformationTranslation. Sorry for that commit it is very huge.
   
   ## Verifying this change
   
   Added multiple new tests for the new operators and for the adapter simulation. In general, all Sink IT cases also verify that the Sink V2 works.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3b172e6de001227e6e383b5e7431eed5b529f8be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389) 
   * 3160e1d83687a15d9a602922aecc2d38f6891afa UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796536914



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;

Review comment:
       Good point. I hope I resolved the situation by removing the ctor of `CommittableCollector ` and moved the `subtaskId` and `numberOfSubtasks` to the `CommittableCollectorSerializer` that is always created once for the committer operator. So the arbitrary values are not part of the `CommittableCollector` anymore. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795469019



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableManager.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CommittableManager<CommT> {

Review comment:
       The idea here is to make the difference between batch and streaming execution mode clear. The `CommittableManager` interface is intended to be used on `endOfInput` and the `CheckpointCommittableManager` interface is used for `notifyCheckpointComplete`. I agree currently it is implemented by the same class but in the end, from an operator perspective, it does not matter.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796778049



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       Hopefully, I found a final solution now I had to change how the parallelism is passed to the transformations 06eb49a. Now defaulting to the env parallelism is the responsibility of the translators and by default, the parallelism is always `ExecutionConfig#DEFAULT_PARALLELISM`. This allows to do:
   
   ```java
                   if (subTransformation.getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT) {
                       if (transformation.getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT) {
                           subTransformation.setParallelism(
                                   context.getStreamGraph().getExecutionConfig().getParallelism());
                       } else {
                           subTransformation.setParallelism(transformation.getParallelism());
                       }
                   }
   
   ```
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3b172e6de001227e6e383b5e7431eed5b529f8be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389) 
   * 3160e1d83687a15d9a602922aecc2d38f6891afa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419) 
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795477843



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link SinkWriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends CommittableManager<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (CommittableManager<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittableManager<CommT> manager :

Review comment:
       I tried your suggestion unfortunately it feels like it complicates the solution. First I also need to pass the committer to the commit method
   ```
   List<CommittableMessage> commit(int subtaskId, int totalSubtasks, long lastestCompletedCheckpointId, Committer<CommT> committer);
   ```
   but the biggest problem is that I need to create the `CommittableSummary`s for every checkpoint so I cannot simply return a list of `CommittableMessage`s. The `CommittableManager` is the only object that holds all necessary information to build the correct summary.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795715352



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {

Review comment:
       We already have a `CommittableManager` that denotes committables during batch execution. Maybe `CommittableCoordinator` is a better name although coordinator is also taken inside of Flink. I am afraid most general terms already have meaning in the code-base somewhere. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1024526229


   Very thanks @fapaul for the update and explanation!
   
   For the `execute` method, sorry I has some misunderstandings initially that the clear happens before compile happens, but it should indeed be happen after the generate() is called. However, there seems still cases that the transformation is not cleared, like users call `StreamExecutionEnvironment#getExecutionPlan()` ? 
   
   
   
   For the Batch exchange, I think perhaps with branches (namely pass the runtime mode to SinkExpander and avoid create the failover region for streaming mode), we could avoid creating the batch exchange in advance for streaming mode, thus we won't need to rely on the behavior of the blocking edge under streaming mode~?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62e100ee426371e3cf041d52fcbf7d5b09af63d3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566) 
   * b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #18428: [Flink 25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017676783


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit f799eeea34b4d3c5691cc4b77f5be103ef1c065c (Thu Jan 20 16:19:51 UTC 2022)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
    * **Invalid pull request title: No valid Jira ID provided**
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1021113345


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r792478403



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittables.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+
+/**
+ * This interface adds checkpoint meta information to the committable.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CheckpointCommittables<CommT> extends Committables<CommT> {

Review comment:
       The idea behind the two interfaces is to have one abstraction for batch execution `CommittableManager` that does not have a checkpoint Id because there a no checkpoint and `CheckpointCommittableManager` for streaming use-cases where checkpoints are supported.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794298730



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittables;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.Committables;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link WriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends Committables<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (Committables<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittables<CommT> checkpoint :
+                committableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId)) {
+            // wait for all committables of the current checkpoint before submission

Review comment:
       I guess this one is related to the `fullyReceived` discussion. The idea is that we do not want to send out committables of the latest checkpoint if we haven't received all of them. Does this make sense?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b3cd4cde01b0207349df086beb909b000b5e0bdd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208) 
   * 2a826e83de723921dd06636571743486cdef432b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1023219238


   thanks for your first review @JingGe I have addressed all your comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3705b617ac596bd4be08fd9ef2e4db40bef586f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154) 
   * b3cd4cde01b0207349df086beb909b000b5e0bdd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791937510



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    boolean hasReceivedAll() {
+        return getNumCommittables() == summary.getNumberOfCommittables();
+    }
+
+    int getNumCommittables() {
+        return requests.size() + numDrained + numFailed;
+    }
+
+    int getNumPending() {
+        return summary.getNumberOfCommittables() - (numDrained + numFailed);
+    }
+
+    int getNumFailed() {
+        return numFailed;
+    }
+
+    boolean isFinished() {
+        return getNumPending() == 0;
+    }
+
+    Stream<CommitRequestImpl<CommT>> getPendingRequests() {
+        return requests.stream().filter(c -> !c.isFinished());
+    }
+
+    List<CommittableWithLineage<CommT>> drainCommitted() {

Review comment:
       I think java doc is required for this method. Will this method be called multiple times? How to make the numbers being synched with the `summary`, i.e. `hasReceivedAll()`? 
   
   There are many numbers defined in this class that need more detailed description about the relationship between them:
   1. total number of commitables in `summary`
   2. total number of `CommitRequestImpl` in `request`
      2.1 number of unfinished request
      2.2 number of failed request
      2.3 number of request that will be drained and removed
   3. numFailed, accumulated if this method would be called multiple time.
   4. numDrained, accumulated if this method would be called multiple time.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795698138



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       I had another look and it might be easier than we initially thought. By default, all transformations have specific default parallelism [1]. So I can simply do the following 
   
   ```java
                   if (subTransformation.getParallelism()
                                   == context.getStreamGraph().getExecutionConfig().getParallelism()
                           && subTransformation.getParallelism() != transformation.getParallelism()) {
                       subTransformation.setParallelism(transformation.getParallelism());
                   }
   
   
   ```
   
   [1] https://github.com/apache/flink/blob/cd8ea8d5b207569f68acc5a3c8db95cd2ca47ba6/flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java#L87




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533) 
   * 62e100ee426371e3cf041d52fcbf7d5b09af63d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027285397


   > Hi Fabian @fapaul ~ Very thanks for the update! As a whole I'm also agree with that we could move some issues to separate PRs to unblock the following tasks, especially those issues only related to the internal implementation.
   > 
   > Currently some main remaining issues from my side are as follows:
   > 
   > **1. About setParallelism:**
   > 
   > I think the current solution to the setParallelism issue is creative, but since it affects the result of `Transformation#getParallelism`, I'm a bit concern on if we have enumerate all the cases. Currently there are two possible issues from my side:
   > 
   > * The semantics of `DataStream#getParallelism` is changed: previously it would always not return -1. This might be fixed by return the env's default if the transformation's parallelism is -1.
   > * `TableOperatorWrapperGenerator#calcParallelismAndResource` might have issues if it deals with both transformations with default parallelism and configured parallelism. Sorry I do not see an easy solution to solve this issue.
   
   I am not really familiar with this part of the code base but so far I did not see any failing tests regarding the `TableOperatorWrapperGenerator` can you elaborate a bit more why the current solution may cause problems.
   
   
   > I tried to see if we could first workaround this issue, but sorry I still not be able to find solutions.
   > 
   > **2. About the repeat transformation**
   > 
   > For the repeat transformation problem, I'm a bit hesitate that why we not remove the newly added transformations instead of the `SinkTransformation`? For example, it is feasible for users to first call `getExecutionPlan()` and then continue to change the properties of the `SinkTransformation`...
   
   Good point I take a look at this tomorrow.
   
   > **3. About the commit protocol**
   > 
   > Currently I'm a bit concern if we should keep there are only one summary message for each (checkpoint, subtask) pair, or we allow multiple messages but they must have the save total count. The problem here is that if we allow update the total count, the following operators would not have a method to know if it has received all the committables for one checkpoint.
   > 
   > Is it possible for now we first throws exception if the `CommitOperator` received the repeat summary message, but keeps the current implementation that it is still possible to emit multiple summary message for a single checkpoint? This poses stricter requirements for `pre-committer toplogy` and `post-committer topology`, thus it would not cause compatibility problem even we change the behavior afterwards.
   > 
   > Also I have tried to think whether unaligned checkpoint + rescaling might cause problems, but currently I do not find issues.
   
   Afaict the motivation to update the commit summary is exactly for unaligned checkpointing. In case the checkpoint barrier overtakes the records we only emit an incomplete summary and later need to emit a new summary with the updated amount of seen records. 
   In general, I am also open to restricting it in the first version and figuring out the unaligned checkpoint use cases later. WDYT?
   
   > 
   > **4. About the PartitionTransformation (Optional)**
   > 
   > > Your suggestion is definitely possible but it does not work for exchanges in the sink topology provided by the user. In this case, I would need to move the current logic from PartitionTransformationTranslator to SinkTransformationTranslator. Perhaps  it is a more general issue and the fix should reside in PartitionTransformationTranslator rather than only in the sink translation.
   > > Is there a downside when adding it to the PartitionTransformationTranslator?
   > 
   > I think changing the partition type on-the-fly might cover the possible issues on the callers. I think `PartitionTransformationTranslator` could check and throw exception for impossible cases, but he should not modify it directly. The callers should ensure to avoid the impossible cases. This issue might be postponed since it only related to the implementation for now.
   
   I think at some point we have to modify the partition transformation. Users might use a partition transformation in their custom sink topologies and they want it to work for batch and streaming mode. So it would be great to allow blocking exchanges in batch and default in streaming mode. 
   Throwing an exception would mean to either support batch or stream. Overall, I agree we can revisit this topic after this PR.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793612274



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface Committables<CommT> {
+    /** Returns a summary of the current commit progress. */
+    CommittableSummary<CommT> getSummary();
+
+    /**
+     * Commits all due committables.
+     *
+     * @param fullyReceived only commit committables if all committables of this checkpoint for a

Review comment:
       Do you have a suggestion here? I am having a hard time to find a better phrasing.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793649807



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       I have the same issue like you have. It seems there is another PR (#18504) will take the responsibility for it. FYI: https://github.com/apache/flink/pull/18333#discussion_r791587285




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791758548



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       It would be great if the changes w.r.t. the ArchUnit do not mix up with this PR




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794777402



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       It seems we indeed do not have a good method to check if parallelism is set. Logically we should have the priority with `customized > sink transformations' settings > env default`. We might have to support it since for compaction we need to set the compact coordinator to parallelism 1...
   
   One possible solution is still to use a separate environment with the default parallelism set to sink transformation's settings... 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       It seems we indeed do not have a good method to check if parallelism is set. Logically we should have the priority with `customized > sink transformations' settings > env default`. We might have to support it since for compaction we need to set the compact coordinator to parallelism 1...
   
   One possible solution is to use a separate environment with the default parallelism set to sink transformation's settings... 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794382729



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       Is there a good way to determine that the user explicitly set the parallelism? So far I assumed that if the user sets the parallelism of the `SinkTransformation` it applies to all transformations that are expanded.
   
   I also do not see a good a way to distinguish between "user set" parallelism because afaik when a `Transformation` is created you always have to pass a parallelism.
   
   In end, we probably have to define our own interpretation of what setting the parallelism at the `SinkTransformation` implies. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793602829



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterStateHandler.java
##########
@@ -17,34 +17,27 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.runtime.state.StateInitializationContext;
-import org.apache.flink.util.function.FunctionWithException;
-
-import java.io.Serializable;
-import java.util.List;
 
 /**
  * Manages the state of a {@link org.apache.flink.api.connector.sink.SinkWriter}. There are only two
  * flavors: stateless handled by {@link StatelessSinkWriterStateHandler} and stateful handled with
  * {@link StatefulSinkWriterStateHandler}.
  *
- * @param <WriterStateT>
+ * @param <InputT> the input type
  */
-interface SinkWriterStateHandler<WriterStateT> extends Serializable {
-    /**
-     * Extracts the writer state from the {@link StateInitializationContext}. The state will be used
-     * to create the writer.
-     */
-    List<WriterStateT> initializeState(StateInitializationContext context) throws Exception;
+interface SinkWriterStateHandler<InputT> {
 
     /**
-     * Stores the state of the supplier. The supplier should only be queried once.
+     * Stores the state of the writer.
      *
-     * @param stateExtractor
-     * @param checkpointId
+     * @param checkpointId the checkpointId
      */
-    void snapshotState(
-            FunctionWithException<Long, List<WriterStateT>, Exception> stateExtractor,
-            long checkpointId)
+    void snapshotState(long checkpointId) throws Exception;
+
+    /** Creates a writer, potentially using state from {@link StateInitializationContext}. */
+    SinkWriter<InputT> createWriter(InitContext initContext, StateInitializationContext context)

Review comment:
       This is the hidden magic method that neither the class name nor the java doc has described. I would suggest using a more feasible class name to cover this functionality.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794446933



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/WriterOperator.java
##########
@@ -56,105 +59,64 @@
  * same parallelism or send them downstream to a {@link CommitterOperator} with a different
  * parallelism.
  *
- * <p>The operator may be part of a sink pipeline and is the first operator. There are currently two
- * ways this operator is used:
- *
- * <ul>
- *   <li>In streaming mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
+ * <p>The operator may be part of a sink pipeline and is the first operator.
  *
  * @param <InputT> the type of the committable
  * @param <CommT> the type of the committable (to send to downstream operators)
- * @param <WriterStateT> the type of the writer state for stateful sinks
  */
-class SinkOperator<InputT, CommT, WriterStateT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<InputT, byte[]>, BoundedOneInput {
+class WriterOperator<InputT, CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>

Review comment:
       I renamed to `SinkWriterOperator` to be consistent.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794485945



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.serialization.SerializationSchema.InitializationContext;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.Sink.ProcessingTimeService;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.StatefulSink;
+import org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter;
+import org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/** Translates Sink V1 into Sink V2. */
+@Internal
+public class SinkV1Adapter<InputT, CommT, WriterStateT, GlobalCommT> implements Sink<InputT> {
+
+    private final org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+            sink;
+
+    private SinkV1Adapter(
+            org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                    sink) {
+        this.sink = sink;
+    }
+
+    public static <InputT> Sink<InputT> wrap(
+            org.apache.flink.api.connector.sink.Sink<InputT, ?, ?, ?> sink) {
+        return new SinkV1Adapter<>(sink).asSpecializedSink();
+    }
+
+    @Override
+    public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+            throws IOException {
+        org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                sink.createWriter(new InitContextAdapter(context), Collections.emptyList());
+        return new SinkWriterV1Adapter<>(writer);
+    }
+
+    public Sink<InputT> asSpecializedSink() {
+        boolean stateful = false;
+        boolean globalCommitter = false;
+        boolean committer = false;
+        if (sink.getWriterStateSerializer().isPresent()) {
+            stateful = true;
+        }
+        if (sink.getGlobalCommittableSerializer().isPresent()) {
+            globalCommitter = true;
+        }
+        if (sink.getCommittableSerializer().isPresent()) {
+            committer = true;
+        }
+
+        if (globalCommitter && committer && stateful) {
+            return new StatefulGlobalTwoPhaseCommittingSinkAdapter();
+        }
+        if (globalCommitter) {
+            return new GlobalCommittingSinkAdapter();
+        }
+        if (committer && stateful) {
+            return new StatefulTwoPhaseCommittingSinkAdapter();
+        }
+        if (committer) {
+            return new TwoPhaseCommittingSinkAdapter();
+        }
+        if (stateful) {
+            return new StatefulSinkAdapter();
+        }
+        return this;
+    }
+
+    private static class SinkWriterV1Adapter<InputT, CommT, WriterStateT>
+            implements StatefulSinkWriter<InputT, WriterStateT>,
+                    PrecommittingSinkWriter<InputT, CommT> {
+
+        private final org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                writer;
+        private boolean endOfInput = false;
+        private final WriterContextAdapter contextAdapter = new WriterContextAdapter();
+
+        public SinkWriterV1Adapter(
+                org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                        writer) {
+            this.writer = writer;
+        }
+
+        @Override
+        public void write(InputT element, Context context)
+                throws IOException, InterruptedException {
+            contextAdapter.setContext(context);
+            this.writer.write(element, contextAdapter);
+        }
+
+        @Override
+        public void flush(boolean endOfInput) throws IOException, InterruptedException {
+            this.endOfInput = endOfInput;
+        }
+
+        @Override
+        public List<WriterStateT> snapshotState(long checkpointId) throws IOException {
+            return writer.snapshotState(checkpointId);
+        }
+
+        @Override
+        public Collection<CommT> prepareCommit() throws IOException, InterruptedException {
+            return writer.prepareCommit(endOfInput);
+        }
+
+        @Override
+        public void close() throws Exception {
+            writer.close();
+        }
+
+        @Override
+        public void writeWatermark(Watermark watermark) throws IOException, InterruptedException {
+            writer.writeWatermark(watermark);
+        }
+    }
+
+    private static class WriterContextAdapter implements SinkWriter.Context {
+        private org.apache.flink.api.connector.sink2.SinkWriter.Context context;
+
+        public void setContext(org.apache.flink.api.connector.sink2.SinkWriter.Context context) {
+            this.context = context;
+        }
+
+        @Override
+        public long currentWatermark() {
+            return context.currentWatermark();
+        }
+
+        @Override
+        public Long timestamp() {
+            return context.timestamp();
+        }
+    }
+
+    private static class InitContextAdapter
+            implements org.apache.flink.api.connector.sink.Sink.InitContext {
+
+        private final InitContext context;
+
+        public InitContextAdapter(InitContext context) {
+            this.context = context;
+        }
+
+        @Override
+        public UserCodeClassLoader getUserCodeClassLoader() {
+            return context.getUserCodeClassLoader();
+        }
+
+        @Override
+        public MailboxExecutor getMailboxExecutor() {
+            return context.getMailboxExecutor();
+        }
+
+        @Override
+        public ProcessingTimeService getProcessingTimeService() {
+            return new ProcessingTimeServiceAdapter(context.getProcessingTimeService());
+        }
+
+        @Override
+        public int getSubtaskId() {
+            return context.getSubtaskId();
+        }
+
+        @Override
+        public int getNumberOfParallelSubtasks() {
+            return context.getNumberOfParallelSubtasks();
+        }
+
+        @Override
+        public SinkWriterMetricGroup metricGroup() {
+            return context.metricGroup();
+        }
+
+        @Override
+        public OptionalLong getRestoredCheckpointId() {
+            return context.getRestoredCheckpointId();
+        }
+
+        public InitializationContext asSerializationSchemaInitializationContext() {
+            return context.asSerializationSchemaInitializationContext();
+        }
+    }
+
+    private static class ProcessingTimeCallbackAdapter implements ProcessingTimeCallback {
+
+        private final ProcessingTimeService.ProcessingTimeCallback processingTimerCallback;
+
+        public ProcessingTimeCallbackAdapter(
+                ProcessingTimeService.ProcessingTimeCallback processingTimerCallback) {
+            this.processingTimerCallback = processingTimerCallback;
+        }
+
+        @Override
+        public void onProcessingTime(long time) throws IOException, InterruptedException {
+            processingTimerCallback.onProcessingTime(time);
+        }
+    }
+
+    private static class ProcessingTimeServiceAdapter implements ProcessingTimeService {
+
+        private final org.apache.flink.api.common.operators.ProcessingTimeService
+                processingTimeService;
+
+        public ProcessingTimeServiceAdapter(
+                org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService) {
+            this.processingTimeService = processingTimeService;
+        }
+
+        @Override
+        public long getCurrentProcessingTime() {
+            return processingTimeService.getCurrentProcessingTime();
+        }
+
+        @Override
+        public void registerProcessingTimer(
+                long time, ProcessingTimeCallback processingTimerCallback) {
+            processingTimeService.registerTimer(
+                    time, new ProcessingTimeCallbackAdapter(processingTimerCallback));
+        }
+    }
+
+    private static class CommitterAdapter<CommT> implements Committer<CommT> {
+
+        private final org.apache.flink.api.connector.sink.Committer<CommT> committer;
+
+        public CommitterAdapter(org.apache.flink.api.connector.sink.Committer<CommT> committer) {
+            this.committer = committer;
+        }
+
+        @Override
+        public void commit(Collection<CommitRequest<CommT>> commitRequests)
+                throws IOException, InterruptedException {
+            List<CommT> failed =
+                    committer.commit(
+                            commitRequests.stream()
+                                    .map(CommitRequest::getCommittable)
+                                    .collect(Collectors.toList()));
+            if (!failed.isEmpty()) {
+                Set<CommT> indexed = Collections.newSetFromMap(new IdentityHashMap<>());
+                indexed.addAll(failed);
+                commitRequests.stream()
+                        .filter(request -> indexed.contains(request.getCommittable()))
+                        .forEach(CommitRequest::retryLater);
+            }
+        }
+
+        @Override
+        public void close() throws Exception {
+            committer.close();
+        }
+    }
+
+    /** Main class to simulate SinkV1 with SinkV2. */
+    @VisibleForTesting
+    @Internal
+    public class PlainSinkAdapter implements Sink<InputT> {
+        @Override
+        public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+                throws IOException {
+            return SinkV1Adapter.this.createWriter(context);
+        }
+
+        @VisibleForTesting
+        public org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                getSink() {
+            return sink;
+        }
+    }
+
+    private class StatefulSinkAdapter extends PlainSinkAdapter
+            implements StatefulSink<InputT, WriterStateT> {
+        @Override
+        public StatefulSinkWriter<InputT, WriterStateT> restoreWriter(
+                InitContext context, Collection<WriterStateT> recoveredState) throws IOException {
+            org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                    sink.createWriter(
+                            new InitContextAdapter(context), new ArrayList<>(recoveredState));
+            return new SinkWriterV1Adapter<>(writer);
+        }
+
+        @Override
+        public SimpleVersionedSerializer<WriterStateT> getWriterStateSerializer() {
+            return sink.getWriterStateSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+    }
+
+    private class TwoPhaseCommittingSinkAdapter extends PlainSinkAdapter
+            implements TwoPhaseCommittingSink<InputT, CommT>, WithCompatibleState {
+        @Override
+        public Committer<CommT> createCommitter() throws IOException {
+            return new CommitterAdapter<>(
+                    sink.createCommitter().orElse(new SinkV1Adapter.NoopCommitter<>()));
+        }
+
+        @Override
+        public SimpleVersionedSerializer<CommT> getCommittableSerializer() {
+            return sink.getCommittableSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+
+        @Override
+        public Collection<String> getCompatibleWriterStateNames() {
+            return sink.getCompatibleStateNames();
+        }
+    }
+
+    private class GlobalCommittingSinkAdapter extends TwoPhaseCommittingSinkAdapter
+            implements WithPostCommitTopology<InputT, CommT> {
+
+        @Override
+        public void addPostCommitTopology(DataStream<CommittableMessage<CommT>> committables) {
+            StandardSinkTopologies.addGlobalCommitter(committables, GlobalCommitterAdapter::new);
+        }
+    }
+
+    private class StatefulTwoPhaseCommittingSinkAdapter extends StatefulSinkAdapter

Review comment:
       AFAICT I need multiple classes or it increases the complexity to implement the given use cases with one class i.e. converting a previously only SinkWriter -> GlobalCommitter topologies now becomes SinkWriter -> ForwardCommitter -> GlobalCommitter. I think the conditional logic is then distributed across all creation methods (i.e. SinkWriter, Commiter, GlobalCommiter) which I find harder to understand instead of doing the checks once and returning different implementations.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink 25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a826e83de723921dd06636571743486cdef432b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794275274



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/PartitionTransformationTranslator.java
##########
@@ -70,13 +73,15 @@
 
         List<Integer> resultIds = new ArrayList<>();
 
+        StreamExchangeMode exchangeMode = transformation.getExchangeMode();
+        if (!supportsBatchExchange && exchangeMode == StreamExchangeMode.BATCH) {
+            exchangeMode = StreamExchangeMode.UNDEFINED;

Review comment:
       Basically in streaming mode, there is no handling for `BATCH` exchange mode they are only relevant in batch execution mode. Therefore it is safe to reset the edges in streaming mode. I'll add a comment




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3160e1d83687a15d9a602922aecc2d38f6891afa Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419) 
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1021113345


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b3cd4cde01b0207349df086beb909b000b5e0bdd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208) 
   * 2a826e83de723921dd06636571743486cdef432b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795698138



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       I had another look and it might be easier than we initially thought. By default, all transformations have specific default parallelism [1]. So I can simply do the following 
   
   ```java
   if (subTransformation.getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT) {
       subTransformation.setParallelism(transformation.getParallelism());
   }
   
   ```
   As soon as users set the parallelism, it won't change during the translation. Another important point is that the `sink transformation` parallelism is equal to the `env default` parallelism. [2]
   
   [1] https://github.com/apache/flink/blob/cd8ea8d5b207569f68acc5a3c8db95cd2ca47ba6/flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java#L87
   [2] https://github.com/fapaul/flink/blob/e135c4f6208fb5efe5c0c0640f90d82f588e244d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java#L73




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796455508



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;

Review comment:
       Hi Fabian~ Thanks for the explanation! My main concern comes from that I think current the `CheckpointCommittableManagerImpl` takes two roles: the first is the manager of committables of one checkpoint in memory and another is the state to be stored in the state. For the first role, having `subtaskId` and `numberOfSubtassks` is natural. But for the second role, the two fields are not needed, currently we set it with arbitrary values (0 and 1 respectively), I think this might cause a bit confusion. Thus it seems to me it might be better if we either extract out the two optional fields, or we do not reuse the same object for these two roles~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617",
       "triggerID" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a881df87350ecd876491656010950c33c61e0cda",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1027956640",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "1027956640",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "a881df87350ecd876491656010950c33c61e0cda",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a881df87350ecd876491656010950c33c61e0cda",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * a881df87350ecd876491656010950c33c61e0cda UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796878192



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -20,249 +20,258 @@
 
 import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getInputStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
+        // Remove the SinkTransformation to not translate it again
+        transformation
+                .getInputStream()
+                .getExecutionEnvironment()
+                .getTransformations()
+                .remove(transformation);
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new SinkWriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new SinkWriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <I> DataStream<I> addFailOverRegion(DataStream<I> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(

Review comment:
       We might need to consider the cases like pre-commit operators starts with an operator with different parallelism. Perhaps we need to add a new identity map here. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1025891089


   > For the `execute` method, sorry I has some misunderstandings initially that the clear happens before compile happens, but it should indeed be happen after the generate() is called. However, there seems still cases that the transformation is not cleared, like users call `StreamExecutionEnvironment#getExecutionPlan()` ?
   
   Afaict `getExecutionPlan` is only used within tests and it feels to me like it is some part of Flink's legacy and is not used anymore. I will double-check that.
   
   > For the Batch exchange, I think perhaps with branches (namely pass the runtime mode to SinkExpander and avoid create the failover region for streaming mode), we could avoid creating the batch exchange in advance for streaming mode, thus we won't need to rely on the behavior of the blocking edge under streaming mode~?
   
   Your suggestion is definitely possible but it does not work for exchanges in the sink topology provided by the user. In this case, I would need to move the current logic from `PartitionTransformationTranslator` to `SinkTransformationTranslator`. Perhaps it is a more general issue and the fix should reside in `PartitionTransformationTranslator` rather than only in the sink translation.
   Is there a downside when adding it to the `PartitionTransformationTranslator`?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794322437



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManager.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Manages the committables coming from one subtask. */
+class SubtaskCommittableManager<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittableManager(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittableManager(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        checkState(
+                requests.size() != summary.getNumberOfCommittables(),

Review comment:
       Might change `!=` to `<` ?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;

Review comment:
       As a whole, perhaps we could remove `subtaskId` and `numberOfSubtasks` from the commit states?
   
   Currently it seems the `subtaskId` and `numberOfSubtasks` is mainly used to create the summary message, and the two values should refers to the index and parallelism of the current committer operator. We might pass the two fields when committing. This would also avoid the case we ignore the two fields (like set them with 0 and 1). 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link SinkWriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends CommittableManager<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (CommittableManager<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittableManager<CommT> manager :

Review comment:
       Is it possible we avoid access `CheckpointCommittableManager` in `CommitterOperator` ?
   
   Currently the states is organized into three levels, namely `commit state -> per-checkpoint state -> per-checkpoint per-subtask state`, perhaps it is better to not access the inner level directly. Here we might have a method in `CommittableCollector` like
   
   ```
   List<CommittableMessage> commit(int subtaskId, int totalSubtasks);
   ```
   
   that commits all the committables and returns the list of messages to emit. 
   
   This might also related to 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManager.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Manages the committables coming from one subtask. */
+class SubtaskCommittableManager<CommT> {
+    private CommittableSummary<CommT> summary;

Review comment:
       Could we directly use a `int numCommittables` or `int numTotal` to bookkeep the total number of the committables? 
   
   The only usage for other fields is the `drainCommitted` method below. I'm a bit concern on the right `subtaskId` to fill in here: if we want to keep the same protocol for the post-committer topology, the subtask id here should be the id of the current commit operator subtask? (Assume users might create operators in pre-committer topology that ends with a non-forward connection, or there is rescale of committer operator after restarting). Thus the subtaskId should be updated, and they could be passed from the `CheckpointManager`, which could also avoid replicate information between different level of state class~

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {

Review comment:
       Is it possible we also change this class to `CommittableManager`? Since `Collector` seems to already have another meaning (something like output) in Flink project and also it would be consistency with the other two levels. 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManager.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Manages the committables coming from one subtask. */
+class SubtaskCommittableManager<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittableManager(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittableManager(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        checkState(
+                requests.size() != summary.getNumberOfCommittables(),
+                "Already received all committables.");
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    /**
+     * Returns whether the received number of committables matches the expected number.
+     *
+     * @return boolean

Review comment:
       The message might need update or remove, also similar to the following methods~

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableManager.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CommittableManager<CommT> {

Review comment:
       Currently it seems `CommittableManager` and `CheckpointCommittableManager` has only one implementation?
   
   Since they are all internal classes, perhaps we do not need to extract the inerfaces here? We might have only one concrete class `CheckpointCommittableManager` that manages the states of the per-checkpoint.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;
+
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /**
+     * For deserialization we can ignore the subtaskId and number of subtasks because it is going to
+     * be merged with another collector that overwrites these values.
+     */
+    CommittableCollector(
+            Map<Long, CheckpointCommittableManagerImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittableManagerImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittableManager<CommT> subtask =

Review comment:
       Similarly, perhaps we could also create the `CommittableWithLinearge` message and use `addMessage` to add them, then we could avoid directly access of the `SubtaskCommittableManager`, which helps to reduce the complexity of the interfaces between different levels. 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittableManagerImpl<CommT> implements CheckpointCommittableManager<CommT> {
+    /** Mapping of subtask id to {@link SubtaskCommittableManager}. */
+    private final Map<Integer, SubtaskCommittableManager<CommT>> subtasksCommittableManagers;
+
+    private final CommittableCollector<CommT> collector;

Review comment:
       We could also remove the reverse dependency if we do not need to maintain the `subtaskId` and the `numberOfTasks`. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] alpreu commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
alpreu commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r789656921



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this.checkpointCommittables = new TreeMap<>(checkpointCommittables);
+        this.subtaskId = 0;
+        this.numberOfSubtasks = 1;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {
+        if (message instanceof CommittableSummary) {
+            addSummary((CommittableSummary<CommT>) message);
+        } else if (message instanceof CommittableWithLineage) {
+            addCommittable((CommittableWithLineage<CommT>) message);
+        }
+    }
+
+    /**
+     * Returns all {@link CheckpointCommittables} until the requested checkpoint id.
+     *
+     * @param checkpointId counter
+     * @return collection of {@link CheckpointCommittables}
+     */
+    public Collection<? extends CheckpointCommittables<CommT>> getCheckpointCommittablesUpTo(
+            long checkpointId) {
+        // clean up fully committed previous checkpoints
+        // this wouldn't work with concurrent unaligned checkpoints
+        Collection<CheckpointCommittablesImpl<CommT>> checkpoints =
+                checkpointCommittables.headMap(checkpointId, true).values();
+        checkpoints.removeIf(CheckpointCommittablesImpl::isFinished);
+        return checkpoints;
+    }
+
+    /**
+     * Returns all {@link CheckpointCommittables} that are currently hold by the collector.
+     *
+     * @return collection of {@link CheckpointCommittables}
+     */
+    @Nullable
+    public Collection<? extends CheckpointCommittables<CommT>> getEndOfInputCommittables() {
+        return getCheckpointCommittablesUpTo(EOI);
+    }
+
+    /**
+     * Returns whether all {@link CheckpointCommittables} currently hold by the collector are either

Review comment:
       ```suggestion
        * Returns whether all {@link CheckpointCommittables} currently held by the collector are either
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793777351



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {

Review comment:
       sure, it is not a big deal. The long name will improve the code readability at the place where the method is called, since code like `addMessage(message)` does not contain enough info. But once the developer get familiar with the code, it is not an issue.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793394852



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {

Review comment:
       Formerly it is not required to have a doc string for `package-private` classes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r790400529



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java
##########
@@ -1240,11 +1238,8 @@ public ExecutionConfig getExecutionConfig() {
             ((InputTypeConfigurable) sinkFunction).setInputType(getType(), getExecutionConfig());
         }
 
-        StreamSink<T> sinkOperator = new StreamSink<>(clean(sinkFunction));
+        DataStreamSink<T> sink = DataStreamSink.forSinkFunction(this, clean(sinkFunction));

Review comment:
       Simplified to `return DataStreamSink.forSinkFunction(this, clean(sinkFunction));` ?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkTransformation.java
##########
@@ -19,49 +19,46 @@ Licensed to the Apache Software Foundation (ASF) under one
 package org.apache.flink.streaming.api.transformations;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.operators.ResourceSpec;
-import org.apache.flink.api.connector.sink.Sink;
-import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.api.java.typeutils.TypeExtractor;
-import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.streaming.api.operators.ChainingStrategy;
 
 import org.apache.flink.shaded.guava30.com.google.common.collect.Lists;
 
 import java.util.Collections;
 import java.util.List;
-import java.util.Optional;
 
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * A {@link Transformation} for {@link Sink}.
  *
  * @param <InputT> The input type of the {@link SinkWriter}
- * @param <CommT> The committable type of the {@link SinkWriter}
- * @param <WriterStateT> The state type of the {@link SinkWriter}
- * @param <GlobalCommT> The global committable type of the {@link
- *     org.apache.flink.api.connector.sink.GlobalCommitter}
+ * @param <OutputT> The output type of the {@link Sink}
  */
 @Internal
-public class SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>
-        extends PhysicalTransformation<Object> {
+public class SinkTransformation<InputT, OutputT> extends PhysicalTransformation<OutputT> {
 
+    private final DataStream<InputT> stream;

Review comment:
       Might rename to `inputStream` and remove the `input` field? The input transformation could be acquired with `inputStream.getTransformation()`. 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java
##########
@@ -1341,7 +1351,7 @@ public ExecutionConfig getExecutionConfig() {
                         serializer,
                         accumulatorName,
                         env.getCheckpointConfig());
-        CollectStreamSink<T> sink = new CollectStreamSink<>(this, factory);
+        DataStreamSink<T> sink = new CollectStreamSink<>(this, factory);

Review comment:
       Is this change necessary~?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -7,7 +7,7 @@
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *     http://www.apache.org/licenses/LICENSE-2.0

Review comment:
       This should not be changed ?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkTransformation.java
##########
@@ -19,49 +19,46 @@ Licensed to the Apache Software Foundation (ASF) under one
 package org.apache.flink.streaming.api.transformations;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.operators.ResourceSpec;
-import org.apache.flink.api.connector.sink.Sink;
-import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.api.java.typeutils.TypeExtractor;
-import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.streaming.api.operators.ChainingStrategy;
 
 import org.apache.flink.shaded.guava30.com.google.common.collect.Lists;
 
 import java.util.Collections;
 import java.util.List;
-import java.util.Optional;
 
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * A {@link Transformation} for {@link Sink}.
  *
  * @param <InputT> The input type of the {@link SinkWriter}
- * @param <CommT> The committable type of the {@link SinkWriter}
- * @param <WriterStateT> The state type of the {@link SinkWriter}
- * @param <GlobalCommT> The global committable type of the {@link
- *     org.apache.flink.api.connector.sink.GlobalCommitter}
+ * @param <OutputT> The output type of the {@link Sink}
  */
 @Internal
-public class SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>
-        extends PhysicalTransformation<Object> {
+public class SinkTransformation<InputT, OutputT> extends PhysicalTransformation<OutputT> {
 
+    private final DataStream<InputT> stream;
+    private final Sink<InputT> sink;
     private final Transformation<InputT> input;
 
-    private final Sink<InputT, CommT, WriterStateT, GlobalCommT> sink;
-
     private ChainingStrategy chainingStrategy;
 
     public SinkTransformation(
+            DataStream<InputT> stream,
+            Sink<InputT> sink,
             Transformation<InputT> input,
-            Sink<InputT, CommT, WriterStateT, GlobalCommT> sink,
+            TypeInformation<OutputT> outputType,
             String name,
             int parallelism) {
-        super(name, TypeExtractor.getForClass(Object.class), parallelism);
+        super(name, outputType, parallelism);
+        this.stream = stream;

Review comment:
       Also add `checkNotNull` for `stream` and `sink` ?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();

Review comment:
       Move this sentence after the check?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/PartitionTransformationTranslator.java
##########
@@ -70,13 +73,15 @@
 
         List<Integer> resultIds = new ArrayList<>();
 
+        StreamExchangeMode exchangeMode = transformation.getExchangeMode();
+        if (!supportsBatchExchange && exchangeMode == StreamExchangeMode.BATCH) {
+            exchangeMode = StreamExchangeMode.UNDEFINED;

Review comment:
       Is this change due to `addFailoverRegion` when translating the sink? I'm a bit tend we not do the check afterwards, instead, it seems better to decide whether to add the failover region according to the runtime mode

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       Also do we consider other properties like `maxParallelism` and `coLocationGroupKey` ?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       We might not update the parallelism if it is set explicitly by users?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {

Review comment:
       `T` would not need to be declared since it is already declared in the class level

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkTransformation.java
##########
@@ -71,47 +68,23 @@ public void setChainingStrategy(ChainingStrategy strategy) {
 
     @Override
     public List<Transformation<?>> getTransitivePredecessors() {
-        final List<Transformation<?>> result = Lists.newArrayList();
-        result.add(this);
-        result.addAll(input.getTransitivePredecessors());
-        return result;
+        return Lists.newArrayList();

Review comment:
       This implementation seems not right? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794371539



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkTransformation.java
##########
@@ -71,47 +68,23 @@ public void setChainingStrategy(ChainingStrategy strategy) {
 
     @Override
     public List<Transformation<?>> getTransitivePredecessors() {
-        final List<Transformation<?>> result = Lists.newArrayList();
-        result.add(this);
-        result.addAll(input.getTransitivePredecessors());
-        return result;
+        return Lists.newArrayList();

Review comment:
       Good catch. I think it is safe to revert this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794708252



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link SinkWriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends CommittableManager<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (CommittableManager<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittableManager<CommT> manager :

Review comment:
       Is it possible we avoid access `CheckpointCommittableManager` in `CommitterOperator` ?
   
   Currently the states is organized into three levels, namely `commit state -> per-checkpoint state -> per-checkpoint per-subtask state`, perhaps it is better to not access the inner level directly. Here we might have a method in `CommittableCollector` like
   
   ```
   List<CommittableMessage> commit(int subtaskId, int totalSubtasks, long lastestCompletedCheckpointId);
   ```
   
   that commits all the committables and returns the list of messages to emit. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r797342100



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -20,249 +20,258 @@
 
 import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getInputStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
+        // Remove the SinkTransformation to not translate it again
+        transformation
+                .getInputStream()
+                .getExecutionEnvironment()
+                .getTransformations()
+                .remove(transformation);
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new SinkWriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new SinkWriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <I> DataStream<I> addFailOverRegion(DataStream<I> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(

Review comment:
       Good point, since all custom topologies are marked as experimental. I would take this as a follow-up.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791758548



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       It would great if the changes w.r.t. the ArchUnit do not mix up with this PR




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796455508



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;

Review comment:
       Hi Fabian~ Thanks for the explanation! My main concern comes from that I think current the `CheckpointCommittableManagerImpl` takes two roles: the first is the manager of committables of one checkpoint in memory and another is the data to be stored in the state. For the first role, having `subtaskId` and `numberOfSubtassks` is natural. But for the second role, the two fields are not needed, currently we set it with arbitrary values (0 and 1 respectively), I think this might cause a bit confusion. Thus it seems to me it might be better if we either extract out the two optional fields, or we do not reuse the same object for these two roles~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586) 
   * e16a9d25a524114b579f05e2872949068588f720 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027672049


   For the parallelism issue, is it possible that 
   1. Before call pre-committer topology, we first modify the default parallelism of env to -1
   2. After the call finished, change the default parallelism back.
   
   Then we could know if the parallelism is set specially, and we then also do not need to change the current behavior of the Transformations ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e16a9d25a524114b579f05e2872949068588f720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595) 
   * ade991fe75b20a8c5403d6d4faa6b03aa105f207 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617",
       "triggerID" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "1027956640",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "a881df87350ecd876491656010950c33c61e0cda",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30627",
       "triggerID" : "1027956640",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "a881df87350ecd876491656010950c33c61e0cda",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30627",
       "triggerID" : "a881df87350ecd876491656010950c33c61e0cda",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * a881df87350ecd876491656010950c33c61e0cda Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30627) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r797436595



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -20,249 +20,259 @@
 
 import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getInputStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private final int environmentParallelism;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.environmentParallelism = executionEnvironment.getParallelism();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            // Reset the environment parallelism temporarily to configure the sub topology
+            // parallelism
+            executionEnvironment.setParallelism(-1);

Review comment:
       Might also use `ExecutionConfig.PARALLELISM_DEFAULT`

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -20,249 +20,259 @@
 
 import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getInputStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private final int environmentParallelism;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.environmentParallelism = executionEnvironment.getParallelism();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            // Reset the environment parallelism temporarily to configure the sub topology
+            // parallelism
+            executionEnvironment.setParallelism(-1);
+
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new SinkWriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
+
+            // Remove all added sink subtransformations to avoid duplications and allow additional
+            // expansions
+            while (executionEnvironment.getTransformations().size() > sizeBefore) {
+                executionEnvironment
+                        .getTransformations()
+                        .remove(executionEnvironment.getTransformations().size() - 1);
+            }
+            // Restore the previous parallelism of the environment
+            executionEnvironment.setParallelism(environmentParallelism);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new SinkWriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <I> DataStream<I> addFailOverRegion(DataStream<I> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <I, R> R adjustTransformations(
+                DataStream<I> inputStream, Function<DataStream<I>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(
+                        subTransformation,
+                        Transformation::getCoLocationGroupKey,
+                        Transformation::setCoLocationGroupKey);
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                if (subTransformation.getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT) {
+                    // The parallelism of the transformation is by default the env parallelism, or
+                    // it is overwritten by the user.
+                    subTransformation.setParallelism(transformation.getParallelism());
+                }
+                if (subTransformation.getMaxParallelism() == -1

Review comment:
       Might change to `subTransformation.getMaxParallelism() < 0 && transformation.getMaxParallelism() > 0`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027956640


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793649807



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       I have the same issue like you have. It seems there is another PR will take the responsibility for it. FYI: https://github.com/apache/flink/pull/18333#discussion_r791587285




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793649807



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       I have the same issue like you have. It seems there is another PR (#18504) will take care of it. FYI: https://github.com/apache/flink/pull/18333#discussion_r791587285




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793772709



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {

Review comment:
       good point, I would suggest to add this info as java doc.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794284695



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java
##########
@@ -40,32 +45,46 @@
 
     private final PhysicalTransformation<T> transformation;
 
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, StreamSink<T> operator) {
-        this.transformation =
-                (PhysicalTransformation<T>)
-                        new LegacySinkTransformation<>(
-                                inputStream.getTransformation(),
-                                "Unnamed",
-                                operator,
-                                inputStream.getExecutionEnvironment().getParallelism());
-    }
-
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, Sink<T, ?, ?, ?> sink) {
-        transformation =
-                (PhysicalTransformation<T>)
-                        new SinkTransformation<>(
-                                inputStream.getTransformation(),
-                                sink,
-                                "Unnamed",
-                                inputStream.getExecutionEnvironment().getParallelism());
+    protected DataStreamSink(PhysicalTransformation<T> transformation) {
+        this.transformation = checkNotNull(transformation);
+    }
+
+    static <T> DataStreamSink<T> forSinkFunction(
+            DataStream<T> inputStream, SinkFunction<T> sinkFunction) {

Review comment:
       This is more like a cleanup that the `DataStreamSink` is the central entry point to add the sink transformation to the datastream. I suspect the `SinkFunction` will be in Flink for a longer time still because it requires Flink 2.0 to remove it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793812428



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface Committables<CommT> {
+    /** Returns a summary of the current commit progress. */
+    CommittableSummary<CommT> getSummary();
+
+    /**
+     * Commits all due committables.
+     *
+     * @param fullyReceived only commit committables if all committables of this checkpoint for a

Review comment:
       I am not sure Whether I understood your idea correctly, for example: 
   
   @param fullyReceived gives the information whether all committables for the checkpoint are received, please refer to {@link CommitterOperator} for more information. Implementation class could use it with its own logic to determine the pending comitRequests that need to be committed, for example to check further if subtasksCommittableManager has received all commitables.   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794763092



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -7,7 +7,7 @@
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *     http://www.apache.org/licenses/LICENSE-2.0

Review comment:
       Very thanks for the explanation~ It seems we have many classes with the inconsistency copyright, thus perhaps we fix it in a separate PR ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #18428: [Flink 25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   * 3705b617ac596bd4be08fd9ef2e4db40bef586f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796590311



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableManager.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CommittableManager<CommT> {

Review comment:
       Hi @fapaul~ Sorry I might not fully got here: It seems currently we are using the method that treats the endOfInput as the checkpoint with id = Long.MAX_VALUE in the implementation? It seems to me that it is sufficient to handle both the streaming and batch cases~? Do we expected to have another `CommittableManager` implementation in the future~?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617",
       "triggerID" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a881df87350ecd876491656010950c33c61e0cda",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30627",
       "triggerID" : "1027956640",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "a881df87350ecd876491656010950c33c61e0cda",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30627",
       "triggerID" : "a881df87350ecd876491656010950c33c61e0cda",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "1027956640",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * a881df87350ecd876491656010950c33c61e0cda Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30627) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r797347655



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -20,249 +20,258 @@
 
 import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getInputStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
+        // Remove the SinkTransformation to not translate it again
+        transformation
+                .getInputStream()
+                .getExecutionEnvironment()
+                .getTransformations()
+                .remove(transformation);
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new SinkWriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new SinkWriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <I> DataStream<I> addFailOverRegion(DataStream<I> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(

Review comment:
       It is also acceptable to me~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027745912


   @gaoyunhaii thanks again for the fruitful discussion. I addressed your concerns
   
   1. Only handling the parallelism overwrite in the SinkTransformationTranslator and deleted the change to the Transformation
   2. Delete the added transformations from the environment after the expansion is finished. It allows to run `getExecutionPlan` and `execute` together and allows further configurations to the sink transformation that still apply to the subtransformations.
   3. For now I disabled the summary updates https://issues.apache.org/jira/browse/FLINK-25920
   4. I left it untouched for now since it also requires followup that is not really related to the Sink


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   * 3705b617ac596bd4be08fd9ef2e4db40bef586f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793585174



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {

Review comment:
       It could actually happen that the `subtaskCommittables` have lower checkpoint Ids because in Flink it can happen that a checkpoint subsumes a previous one.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793787154



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittables.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+
+/**
+ * This interface adds checkpoint meta information to the committable.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CheckpointCommittables<CommT> extends Committables<CommT> {

Review comment:
       fair enough, maybe make sense to point out that the checkpointId provided by CheckpointCommittableManager is not the checkpointId that `CommittableSummary` held by `CommittableManager` provides




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794289252



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterStateHandler.java
##########
@@ -17,34 +17,27 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.runtime.state.StateInitializationContext;
-import org.apache.flink.util.function.FunctionWithException;
-
-import java.io.Serializable;
-import java.util.List;
 
 /**
  * Manages the state of a {@link org.apache.flink.api.connector.sink.SinkWriter}. There are only two
  * flavors: stateless handled by {@link StatelessSinkWriterStateHandler} and stateful handled with
  * {@link StatefulSinkWriterStateHandler}.
  *
- * @param <WriterStateT>
+ * @param <InputT> the input type
  */
-interface SinkWriterStateHandler<WriterStateT> extends Serializable {
-    /**
-     * Extracts the writer state from the {@link StateInitializationContext}. The state will be used
-     * to create the writer.
-     */
-    List<WriterStateT> initializeState(StateInitializationContext context) throws Exception;
+interface SinkWriterStateHandler<InputT> {
 
     /**
-     * Stores the state of the supplier. The supplier should only be queried once.
+     * Stores the state of the writer.
      *
-     * @param stateExtractor
-     * @param checkpointId
+     * @param checkpointId the checkpointId
      */
-    void snapshotState(
-            FunctionWithException<Long, List<WriterStateT>, Exception> stateExtractor,
-            long checkpointId)
+    void snapshotState(long checkpointId) throws Exception;
+
+    /** Creates a writer, potentially using state from {@link StateInitializationContext}. */
+    SinkWriter<InputT> createWriter(InitContext initContext, StateInitializationContext context)

Review comment:
       Can you elaborate on why the name is not descriptive? It creates a sink writer instance. I cannot easily change the class name because it is part of the public API and overall this interface is solely intended for internal usage. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r792447708



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       I guess this change should have belonged to this ticket https://issues.apache.org/jira/browse/FLINK-25570 I am not sure why the tests did not fail before. I can make this change on a separate commit but I would keep it on this PR.
   
   For the other archunit changes afaict they are related to changes of this PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793585174



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {

Review comment:
       I could actually happen that the `subtaskCommittables` have lower checkpoint Ids because in Flink it can happen that a checkpoint subsumes a previous one.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793591197



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommitRequestImpl.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+
+/**
+ * Internal implementation to commit a specific committable and handle the response.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommitRequestImpl<CommT> implements Committer.CommitRequest<CommT> {
+
+    private CommT committable;
+    private int numRetries;
+    private CommitRequestState state;
+
+    protected CommitRequestImpl(CommT committable) {
+        this.committable = committable;
+        state = CommitRequestState.RECEIVED;
+    }
+
+    protected CommitRequestImpl(CommT committable, int numRetries, CommitRequestState state) {
+        this.committable = committable;
+        this.numRetries = numRetries;
+        this.state = state;
+    }
+
+    boolean isFinished() {
+        return state.isFinalState();
+    }
+
+    CommitRequestState getState() {
+        return state;
+    }
+
+    @Override
+    public CommT getCommittable() {
+        return committable;
+    }
+
+    @Override
+    public int getNumberOfRetries() {
+        return numRetries;
+    }
+
+    @Override
+    public void signalFailedWithKnownReason(Throwable t) {
+        state = CommitRequestState.FAILED;
+        // add metric later

Review comment:
       I opened a ticket to implement the metrics https://issues.apache.org/jira/browse/FLINK-25857




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794299193



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {
+        this.subtasksCommittables = checkNotNull(subtasksCommittables);
+        // dummy collector, never used
+        this.collector = new CommittableCollector<>(0, 1);
+        this.checkpointId = checkpointId;
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return checkpointId;
+    }
+
+    Collection<SubtaskCommittables<CommT>> getSubtasks() {
+        return subtasksCommittables.values();
+    }
+
+    void addSummary(CommittableSummary<CommT> summary) {

Review comment:
       I renamed it to `upserSummary`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 3b172e6de001227e6e383b5e7431eed5b529f8be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389) 
   * 3160e1d83687a15d9a602922aecc2d38f6891afa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3705b617ac596bd4be08fd9ef2e4db40bef586f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul merged pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul merged pull request #18428:
URL: https://github.com/apache/flink/pull/18428


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791758548



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       I think these should not be mixed up with this PR

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface Committables<CommT> {
+    /** Returns a summary of the current commit progress. */
+    CommittableSummary<CommT> getSummary();
+
+    /**
+     * Commits all due committables.
+     *
+     * @param fullyReceived only commit committables if all committables of this checkpoint for a

Review comment:
       This description seems to be the implementation logic. Instead of it, we should describe the purpose of this parameter at the interface level

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    boolean hasReceivedAll() {
+        return getNumCommittables() == summary.getNumberOfCommittables();
+    }
+
+    int getNumCommittables() {
+        return requests.size() + numDrained + numFailed;
+    }
+
+    int getNumPending() {

Review comment:
       It is difficult to understand the different between `getNumPending` and `getPendingRequests`, since both of them are dealing with committables, one is via `CommittableSummary` and the other is via `CommitRequestImpl`. I didn't find a feasible name for it, therefore I would suggest adding java doc to make it clear.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;

Review comment:
       ```suggestion
       private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittablesMap;
   ```

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {

Review comment:
       checkpointId could be get from `SubtaskCommittables`

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {
+        this.subtasksCommittables = checkNotNull(subtasksCommittables);
+        // dummy collector, never used
+        this.collector = new CommittableCollector<>(0, 1);
+        this.checkpointId = checkpointId;
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return checkpointId;
+    }
+
+    Collection<SubtaskCommittables<CommT>> getSubtasks() {
+        return subtasksCommittables.values();
+    }
+
+    void addSummary(CommittableSummary<CommT> summary) {

Review comment:
       should actually be upsertSummary(...)

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {

Review comment:
       Nit

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittables.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+
+/**
+ * This interface adds checkpoint meta information to the committable.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CheckpointCommittables<CommT> extends Committables<CommT> {

Review comment:
       From OOP point of view, this interface introduced the checkpoint, which means `Committables` must be at a higher level above it and has nothing to do with the checkpoint. But the fact is that `Committables` has already been dealing with `CommittableSummary` and `CommittableWithLineage` that can all provide checkpointId. If it was intended to distinguish between them, more info will be required to describe when to use `Committables` and when to use `CheckpointCommittables`.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {

Review comment:
       java doc

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */

Review comment:
       Java doc is required to describe the deserialization why `subtaskId` and `numberOfSubtasks` are not required for serde. Add reference to the class where it is handled by custom serialization/deserialization, i.e. `CommittableCollectorSerializer`

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    boolean hasReceivedAll() {
+        return getNumCommittables() == summary.getNumberOfCommittables();
+    }
+
+    int getNumCommittables() {
+        return requests.size() + numDrained + numFailed;
+    }
+
+    int getNumPending() {
+        return summary.getNumberOfCommittables() - (numDrained + numFailed);
+    }
+
+    int getNumFailed() {
+        return numFailed;
+    }
+
+    boolean isFinished() {
+        return getNumPending() == 0;
+    }
+
+    Stream<CommitRequestImpl<CommT>> getPendingRequests() {
+        return requests.stream().filter(c -> !c.isFinished());
+    }
+
+    List<CommittableWithLineage<CommT>> drainCommitted() {
+        List<CommittableWithLineage<CommT>> committed = new ArrayList<>(requests.size());
+        for (Iterator<CommitRequestImpl<CommT>> iterator = requests.iterator();
+                iterator.hasNext(); ) {
+            CommitRequestImpl<CommT> request = iterator.next();
+            if (!request.isFinished()) {
+                continue;
+            }
+            if (request.getState() == CommitRequestState.FAILED) {
+                numFailed += 1;

Review comment:
       iterator.remove()? otherwise, failed commitables will be counted twice

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.

Review comment:
       It seems that there are now two meanings of the word "comittables", since the new interface `Committables` has been created. Afaik, "commitables" has been used as a conceptual name previously very often. There is no corresponding domain entity developed for it and just let it open for users to define via the generic type `CommT`.
   
   Since all methods in this interface are working with subclasses of `ComittableMessage`, this interface is actually responsible for managing all commitables and all kinds of `ComittableMessage`, i.e. `ComittableSummary` and `ComittableWithLineage`, which turns out that calling this interface as `ComittablesManager` would be another naming option that is easier to understand the concept behind.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;

Review comment:
       add single line comment for the key -> checkpointId

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;

Review comment:
       ```suggestion
       private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittablesMap;
   ```

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    boolean hasReceivedAll() {
+        return getNumCommittables() == summary.getNumberOfCommittables();
+    }
+
+    int getNumCommittables() {
+        return requests.size() + numDrained + numFailed;
+    }
+
+    int getNumPending() {
+        return summary.getNumberOfCommittables() - (numDrained + numFailed);
+    }
+
+    int getNumFailed() {
+        return numFailed;
+    }
+
+    boolean isFinished() {
+        return getNumPending() == 0;
+    }
+
+    Stream<CommitRequestImpl<CommT>> getPendingRequests() {
+        return requests.stream().filter(c -> !c.isFinished());
+    }
+
+    List<CommittableWithLineage<CommT>> drainCommitted() {

Review comment:
       I think java doc is required for this method. Will this method be called multiple times? How to make the numbers being synched with the `summary`, i.e. `hasReceivedAll()`? Will the failed request be retried?
   
   There many numbers defined in this class that need more detailed description about the relationship between them:
   1. total number of commitables in `summary`
   2. total number of `CommitRequestImpl` in `request`
      2.1 number of unfinished request
      2.2 number of failed request
      2.3 number of request that will be drained and removed
   3. numFailed, accumulated if this method would be called multiple time.
   4. numDrained, accumulated if this method would be called multiple time.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;

Review comment:
       what is the purpose of using `subtaskId`? 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {

Review comment:
       As additional info to other comments: this method opens the door to let the total numbers of request be greater than the total number of commitables in the `summary`.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    boolean hasReceivedAll() {
+        return getNumCommittables() == summary.getNumberOfCommittables();
+    }
+
+    int getNumCommittables() {
+        return requests.size() + numDrained + numFailed;
+    }
+
+    int getNumPending() {
+        return summary.getNumberOfCommittables() - (numDrained + numFailed);
+    }
+
+    int getNumFailed() {
+        return numFailed;
+    }
+
+    boolean isFinished() {
+        return getNumPending() == 0;
+    }
+
+    Stream<CommitRequestImpl<CommT>> getPendingRequests() {
+        return requests.stream().filter(c -> !c.isFinished());
+    }
+
+    List<CommittableWithLineage<CommT>> drainCommitted() {
+        List<CommittableWithLineage<CommT>> committed = new ArrayList<>(requests.size());
+        for (Iterator<CommitRequestImpl<CommT>> iterator = requests.iterator();
+                iterator.hasNext(); ) {
+            CommitRequestImpl<CommT> request = iterator.next();
+            if (!request.isFinished()) {
+                continue;
+            }
+            if (request.getState() == CommitRequestState.FAILED) {
+                numFailed += 1;
+                continue;
+            } else {
+                committed.add(
+                        new CommittableWithLineage<>(
+                                request.getCommittable(),
+                                summary.getCheckpointId().isPresent()
+                                        ? summary.getCheckpointId().getAsLong()
+                                        : null,
+                                summary.getSubtaskId()));
+            }
+            iterator.remove();
+        }
+
+        numDrained += committed.size();
+        return committed;
+    }
+
+    CommittableSummary<CommT> getSummary() {
+        return summary;
+    }
+
+    void setSummary(CommittableSummary<CommT> summary) {
+        this.summary = summary;
+    }
+
+    int getNumDrained() {
+        return numDrained;
+    }
+
+    Deque<CommitRequestImpl<CommT>> getRequests() {
+        return requests;
+    }
+
+    SubtaskCommittables<CommT> merge(SubtaskCommittables<CommT> other) {
+        CommittableSummary<CommT> otherSummary = other.getSummary();
+        checkArgument(otherSummary.getSubtaskId() == this.summary.getSubtaskId());
+        OptionalLong checkpointId = this.summary.getCheckpointId();
+        this.summary =
+                new CommittableSummary<>(
+                        this.summary.getSubtaskId(),
+                        this.summary.getNumberOfCommittables(),

Review comment:
       Should this be the `numberOfSubtasks`?  How does the merge work, if the other has a different numberOfSubtasks? Or should check that the numberOfSubtasks should be equal?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {
+        this.subtasksCommittables = checkNotNull(subtasksCommittables);
+        // dummy collector, never used
+        this.collector = new CommittableCollector<>(0, 1);
+        this.checkpointId = checkpointId;
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return checkpointId;
+    }
+
+    Collection<SubtaskCommittables<CommT>> getSubtasks() {

Review comment:
       It would be better to use a suitable name, since Subtask has its won domain entity meaning. 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;

Review comment:
       // key -> subtaskId

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {

Review comment:
       ```suggestion
       public void addCommittableMessage(CommittableMessage<CommT> message) {
   ```

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {
+        if (message instanceof CommittableSummary) {
+            addSummary((CommittableSummary<CommT>) message);
+        } else if (message instanceof CommittableWithLineage) {
+            addCommittable((CommittableWithLineage<CommT>) message);
+        }
+    }
+
+    /**
+     * Returns all {@link CheckpointCommittables} until the requested checkpoint id.
+     *
+     * @param checkpointId counter
+     * @return collection of {@link CheckpointCommittables}
+     */
+    public Collection<? extends CheckpointCommittables<CommT>> getCheckpointCommittablesUpTo(
+            long checkpointId) {
+        // clean up fully committed previous checkpoints
+        // this wouldn't work with concurrent unaligned checkpoints
+        Collection<CheckpointCommittablesImpl<CommT>> checkpoints =
+                checkpointCommittables.headMap(checkpointId, true).values();
+        checkpoints.removeIf(CheckpointCommittablesImpl::isFinished);
+        return checkpoints;
+    }
+
+    /**
+     * Returns all {@link CheckpointCommittables} that are currently hold by the collector.
+     *
+     * @return collection of {@link CheckpointCommittables}
+     */
+    @Nullable
+    public Collection<? extends CheckpointCommittables<CommT>> getEndOfInputCommittables() {
+        return getCheckpointCommittablesUpTo(EOI);
+    }
+
+    /**
+     * Returns whether all {@link CheckpointCommittables} currently hold by the collector are either
+     * committed or failed.
+     *
+     * @return state of the {@link CheckpointCommittables}
+     */
+    public boolean isFinished() {
+        return checkpointCommittables.values().stream()
+                .allMatch(CheckpointCommittablesImpl::isFinished);
+    }
+
+    /**
+     * Merges all information from an external collector into this collector.
+     *
+     * <p>This method is important during recovery from existing state.
+     *
+     * @param cc other {@link CommittableCollector}
+     */
+    public void merge(CommittableCollector<CommT> cc) {
+        for (Entry<Long, CheckpointCommittablesImpl<CommT>> checkpointEntry :
+                cc.checkpointCommittables.entrySet()) {
+            checkpointCommittables.merge(
+                    checkpointEntry.getKey(),
+                    checkpointEntry.getValue(),
+                    CheckpointCommittablesImpl::merge);
+        }
+    }
+
+    /**
+     * Returns number of subtasks.
+     *
+     * @return number of subtasks
+     */
+    public int getNumberOfSubtasks() {
+        return numberOfSubtasks;
+    }
+
+    /**
+     * Returns subtask id.
+     *
+     * @return subtask id.
+     */
+    public int getSubtaskId() {
+        return subtaskId;
+    }
+
+    /**
+     * Returns a new committable collector that deep copies all internals.
+     *
+     * @return {@link CommittableCollector}
+     */
+    public CommittableCollector<CommT> copy() {
+        return new CommittableCollector<>(
+                checkpointCommittables.entrySet().stream()
+                        .map(e -> Tuple2.of(e.getKey(), e.getValue().copy()))
+                        .collect(Collectors.toMap((t) -> t.f0, (t) -> t.f1)),
+                subtaskId,
+                numberOfSubtasks);
+    }
+
+    Collection<CheckpointCommittablesImpl<CommT>> getCheckpointCommittables() {
+        return checkpointCommittables.values();
+    }
+
+    private void addSummary(CommittableSummary<CommT> summary) {
+        checkpointCommittables
+                .computeIfAbsent(
+                        summary.getCheckpointId().orElse(EOI),
+                        key ->
+                                new CheckpointCommittablesImpl<>(
+                                        this, summary.getCheckpointId().orElse(EOI)))
+                .addSummary(summary);
+    }
+
+    private void addCommittable(CommittableWithLineage<CommT> committable) {
+        getCheckpointCommittables(committable).addCommittable(committable);
+    }
+
+    private CheckpointCommittablesImpl<CommT> getCheckpointCommittables(
+            CommittableMessage<CommT> committable) {

Review comment:
       ```suggestion
               OptionalLong checkpointId) {
   ```

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommitRequestImpl.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+
+/**
+ * Internal implementation to commit a specific committable and handle the response.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommitRequestImpl<CommT> implements Committer.CommitRequest<CommT> {
+
+    private CommT committable;
+    private int numRetries;
+    private CommitRequestState state;
+
+    protected CommitRequestImpl(CommT committable) {
+        this.committable = committable;
+        state = CommitRequestState.RECEIVED;
+    }
+
+    protected CommitRequestImpl(CommT committable, int numRetries, CommitRequestState state) {
+        this.committable = committable;
+        this.numRetries = numRetries;
+        this.state = state;
+    }
+
+    boolean isFinished() {
+        return state.isFinalState();
+    }
+
+    CommitRequestState getState() {
+        return state;
+    }
+
+    @Override
+    public CommT getCommittable() {
+        return committable;
+    }
+
+    @Override
+    public int getNumberOfRetries() {
+        return numRetries;
+    }
+
+    @Override
+    public void signalFailedWithKnownReason(Throwable t) {
+        state = CommitRequestState.FAILED;
+        // add metric later

Review comment:
       ```suggestion
           // TODO(...) add metric later
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791771156



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.

Review comment:
       It seems that there are now two meanings of the word "comittables", since the new interface `Committables` has been created. Afaik, "commitables" has been used as a conceptual name previously very often. There is no corresponding domain entity developed for it and just let it open for users to define via the generic type `CommT`.
   
   Since all methods in this interface are working with subclasses of `ComittableMessage`, this interface is actually responsible for managing all commitables and all kinds of `ComittableMessage`, i.e. `ComittableSummary` and `ComittableWithLineage`, which turns out that calling this interface as `ComittablesManager` would be alternative naming option that is easier to understand the concept behind.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791802037



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittables.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+
+/**
+ * This interface adds checkpoint meta information to the committable.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CheckpointCommittables<CommT> extends Committables<CommT> {

Review comment:
       From OOP point of view, this interface introduced the checkpoint, which means `Committables` must be at a higher level above it and has nothing to do with the checkpoint. But the fact is that `Committables` has already been dealing with `CommittableSummary` and `CommittableWithLineage` that can all provide checkpointId. If it was intended to distinguish between them, more info will be required to describe when to use `Committables` (at a higher abstraction level without the checkpoint concept) and when to use `CheckpointCommittables`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008) 
   * 3705b617ac596bd4be08fd9ef2e4db40bef586f2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3705b617ac596bd4be08fd9ef2e4db40bef586f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154) 
   * b3cd4cde01b0207349df086beb909b000b5e0bdd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796434248



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       Hi Fabian~I'm a bit afraid there might be bad cases for this method since users might
   ```
   env.setParallelism(1);
   DataStream x = ...
   x.addSink(new FinkSink()).setParallism(2)
   ```
   
   while inside the precommit topology, operators like CompactorCoordiantor is set to 1 specially.
   
   In this case, the final parallelism for the compactor coordinator would still be 2 and break the priority?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796849334



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableManager.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CommittableManager<CommT> {

Review comment:
       Thanks for the explanation! I still think it would be better to not introduce the interfaces if it is not used for now, but I do not strongly argue for that for now. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796849334



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableManager.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CommittableManager<CommT> {

Review comment:
       Thanks for the explanation! I still think it would be better to not introduce the interfaces if it is not used for now, but I do not strongly argue for that for now~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794285980



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java
##########
@@ -131,6 +150,10 @@ protected DataStreamSink(DataStream<T> inputStream, Sink<T, ?, ?, ?> sink) {
      */
     @PublicEvolving
     public DataStreamSink<T> setUidHash(String uidHash) {
+        if (!(transformation instanceof LegacySinkTransformation)) {

Review comment:
       No, this exception immediately occurs when the main method is executed before the transformations are translated. It fails the whole job submission.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b3cd4cde01b0207349df086beb909b000b5e0bdd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794276611



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.serialization.SerializationSchema.InitializationContext;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.Sink.ProcessingTimeService;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.StatefulSink;
+import org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter;
+import org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/** Translates Sink V1 into Sink V2. */
+@Internal
+public class SinkV1Adapter<InputT, CommT, WriterStateT, GlobalCommT> implements Sink<InputT> {
+
+    private final org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+            sink;
+
+    private SinkV1Adapter(
+            org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                    sink) {
+        this.sink = sink;
+    }
+
+    public static <InputT> Sink<InputT> wrap(
+            org.apache.flink.api.connector.sink.Sink<InputT, ?, ?, ?> sink) {
+        return new SinkV1Adapter<>(sink).asSpecializedSink();
+    }
+
+    @Override
+    public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+            throws IOException {
+        org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                sink.createWriter(new InitContextAdapter(context), Collections.emptyList());
+        return new SinkWriterV1Adapter<>(writer);
+    }
+
+    public Sink<InputT> asSpecializedSink() {
+        boolean stateful = false;
+        boolean globalCommitter = false;
+        boolean committer = false;
+        if (sink.getWriterStateSerializer().isPresent()) {
+            stateful = true;
+        }
+        if (sink.getGlobalCommittableSerializer().isPresent()) {
+            globalCommitter = true;
+        }
+        if (sink.getCommittableSerializer().isPresent()) {
+            committer = true;
+        }
+
+        if (globalCommitter && committer && stateful) {
+            return new StatefulGlobalTwoPhaseCommittingSinkAdapter();
+        }
+        if (globalCommitter) {
+            return new GlobalCommittingSinkAdapter();
+        }
+        if (committer && stateful) {
+            return new StatefulTwoPhaseCommittingSinkAdapter();
+        }
+        if (committer) {
+            return new TwoPhaseCommittingSinkAdapter();
+        }
+        if (stateful) {
+            return new StatefulSinkAdapter();
+        }
+        return this;
+    }
+
+    private static class SinkWriterV1Adapter<InputT, CommT, WriterStateT>
+            implements StatefulSinkWriter<InputT, WriterStateT>,
+                    PrecommittingSinkWriter<InputT, CommT> {
+
+        private final org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                writer;
+        private boolean endOfInput = false;
+        private final WriterContextAdapter contextAdapter = new WriterContextAdapter();
+
+        public SinkWriterV1Adapter(
+                org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                        writer) {
+            this.writer = writer;
+        }
+
+        @Override
+        public void write(InputT element, Context context)
+                throws IOException, InterruptedException {
+            contextAdapter.setContext(context);
+            this.writer.write(element, contextAdapter);
+        }
+
+        @Override
+        public void flush(boolean endOfInput) throws IOException, InterruptedException {
+            this.endOfInput = endOfInput;
+        }
+
+        @Override
+        public List<WriterStateT> snapshotState(long checkpointId) throws IOException {
+            return writer.snapshotState(checkpointId);
+        }
+
+        @Override
+        public Collection<CommT> prepareCommit() throws IOException, InterruptedException {
+            return writer.prepareCommit(endOfInput);
+        }
+
+        @Override
+        public void close() throws Exception {
+            writer.close();
+        }
+
+        @Override
+        public void writeWatermark(Watermark watermark) throws IOException, InterruptedException {
+            writer.writeWatermark(watermark);
+        }
+    }
+
+    private static class WriterContextAdapter implements SinkWriter.Context {
+        private org.apache.flink.api.connector.sink2.SinkWriter.Context context;
+
+        public void setContext(org.apache.flink.api.connector.sink2.SinkWriter.Context context) {
+            this.context = context;
+        }
+
+        @Override
+        public long currentWatermark() {
+            return context.currentWatermark();
+        }
+
+        @Override
+        public Long timestamp() {
+            return context.timestamp();
+        }
+    }
+
+    private static class InitContextAdapter
+            implements org.apache.flink.api.connector.sink.Sink.InitContext {
+
+        private final InitContext context;
+
+        public InitContextAdapter(InitContext context) {
+            this.context = context;
+        }
+
+        @Override
+        public UserCodeClassLoader getUserCodeClassLoader() {
+            return context.getUserCodeClassLoader();
+        }
+
+        @Override
+        public MailboxExecutor getMailboxExecutor() {
+            return context.getMailboxExecutor();
+        }
+
+        @Override
+        public ProcessingTimeService getProcessingTimeService() {
+            return new ProcessingTimeServiceAdapter(context.getProcessingTimeService());
+        }
+
+        @Override
+        public int getSubtaskId() {
+            return context.getSubtaskId();
+        }
+
+        @Override
+        public int getNumberOfParallelSubtasks() {
+            return context.getNumberOfParallelSubtasks();
+        }
+
+        @Override
+        public SinkWriterMetricGroup metricGroup() {
+            return context.metricGroup();
+        }
+
+        @Override
+        public OptionalLong getRestoredCheckpointId() {
+            return context.getRestoredCheckpointId();
+        }
+
+        public InitializationContext asSerializationSchemaInitializationContext() {
+            return context.asSerializationSchemaInitializationContext();
+        }
+    }
+
+    private static class ProcessingTimeCallbackAdapter implements ProcessingTimeCallback {
+
+        private final ProcessingTimeService.ProcessingTimeCallback processingTimerCallback;
+
+        public ProcessingTimeCallbackAdapter(
+                ProcessingTimeService.ProcessingTimeCallback processingTimerCallback) {
+            this.processingTimerCallback = processingTimerCallback;
+        }
+
+        @Override
+        public void onProcessingTime(long time) throws IOException, InterruptedException {
+            processingTimerCallback.onProcessingTime(time);
+        }
+    }
+
+    private static class ProcessingTimeServiceAdapter implements ProcessingTimeService {
+
+        private final org.apache.flink.api.common.operators.ProcessingTimeService
+                processingTimeService;
+
+        public ProcessingTimeServiceAdapter(
+                org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService) {
+            this.processingTimeService = processingTimeService;
+        }
+
+        @Override
+        public long getCurrentProcessingTime() {
+            return processingTimeService.getCurrentProcessingTime();
+        }
+
+        @Override
+        public void registerProcessingTimer(
+                long time, ProcessingTimeCallback processingTimerCallback) {
+            processingTimeService.registerTimer(
+                    time, new ProcessingTimeCallbackAdapter(processingTimerCallback));
+        }
+    }
+
+    private static class CommitterAdapter<CommT> implements Committer<CommT> {
+
+        private final org.apache.flink.api.connector.sink.Committer<CommT> committer;
+
+        public CommitterAdapter(org.apache.flink.api.connector.sink.Committer<CommT> committer) {
+            this.committer = committer;
+        }
+
+        @Override
+        public void commit(Collection<CommitRequest<CommT>> commitRequests)
+                throws IOException, InterruptedException {
+            List<CommT> failed =
+                    committer.commit(
+                            commitRequests.stream()
+                                    .map(CommitRequest::getCommittable)
+                                    .collect(Collectors.toList()));
+            if (!failed.isEmpty()) {
+                Set<CommT> indexed = Collections.newSetFromMap(new IdentityHashMap<>());
+                indexed.addAll(failed);
+                commitRequests.stream()
+                        .filter(request -> indexed.contains(request.getCommittable()))
+                        .forEach(CommitRequest::retryLater);
+            }
+        }
+
+        @Override
+        public void close() throws Exception {
+            committer.close();
+        }
+    }
+
+    /** Main class to simulate SinkV1 with SinkV2. */
+    @VisibleForTesting
+    @Internal
+    public class PlainSinkAdapter implements Sink<InputT> {
+        @Override
+        public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+                throws IOException {
+            return SinkV1Adapter.this.createWriter(context);
+        }
+
+        @VisibleForTesting
+        public org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                getSink() {
+            return sink;
+        }
+    }
+
+    private class StatefulSinkAdapter extends PlainSinkAdapter
+            implements StatefulSink<InputT, WriterStateT> {
+        @Override
+        public StatefulSinkWriter<InputT, WriterStateT> restoreWriter(
+                InitContext context, Collection<WriterStateT> recoveredState) throws IOException {
+            org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                    sink.createWriter(
+                            new InitContextAdapter(context), new ArrayList<>(recoveredState));
+            return new SinkWriterV1Adapter<>(writer);
+        }
+
+        @Override
+        public SimpleVersionedSerializer<WriterStateT> getWriterStateSerializer() {
+            return sink.getWriterStateSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+    }
+
+    private class TwoPhaseCommittingSinkAdapter extends PlainSinkAdapter
+            implements TwoPhaseCommittingSink<InputT, CommT>, WithCompatibleState {
+        @Override
+        public Committer<CommT> createCommitter() throws IOException {
+            return new CommitterAdapter<>(
+                    sink.createCommitter().orElse(new SinkV1Adapter.NoopCommitter<>()));
+        }
+
+        @Override
+        public SimpleVersionedSerializer<CommT> getCommittableSerializer() {
+            return sink.getCommittableSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+
+        @Override
+        public Collection<String> getCompatibleWriterStateNames() {
+            return sink.getCompatibleStateNames();
+        }
+    }
+
+    private class GlobalCommittingSinkAdapter extends TwoPhaseCommittingSinkAdapter
+            implements WithPostCommitTopology<InputT, CommT> {
+
+        @Override
+        public void addPostCommitTopology(DataStream<CommittableMessage<CommT>> committables) {
+            StandardSinkTopologies.addGlobalCommitter(committables, GlobalCommitterAdapter::new);
+        }
+    }
+
+    private class StatefulTwoPhaseCommittingSinkAdapter extends StatefulSinkAdapter

Review comment:
       I am not sure I understand your idea. The problem is that previously we only had one interface `Sink<InputT, CommT, WriterStateT, GlobalCommT>`  that is now represented by multiple mixins. So I had to split the different use cases into different classes implementing the new interfaces.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794352288



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.serialization.SerializationSchema.InitializationContext;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.Sink.ProcessingTimeService;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.StatefulSink;
+import org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter;
+import org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/** Translates Sink V1 into Sink V2. */
+@Internal
+public class SinkV1Adapter<InputT, CommT, WriterStateT, GlobalCommT> implements Sink<InputT> {
+
+    private final org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+            sink;
+
+    private SinkV1Adapter(
+            org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                    sink) {
+        this.sink = sink;
+    }
+
+    public static <InputT> Sink<InputT> wrap(
+            org.apache.flink.api.connector.sink.Sink<InputT, ?, ?, ?> sink) {
+        return new SinkV1Adapter<>(sink).asSpecializedSink();
+    }
+
+    @Override
+    public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+            throws IOException {
+        org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                sink.createWriter(new InitContextAdapter(context), Collections.emptyList());
+        return new SinkWriterV1Adapter<>(writer);
+    }
+
+    public Sink<InputT> asSpecializedSink() {
+        boolean stateful = false;
+        boolean globalCommitter = false;
+        boolean committer = false;
+        if (sink.getWriterStateSerializer().isPresent()) {
+            stateful = true;
+        }
+        if (sink.getGlobalCommittableSerializer().isPresent()) {
+            globalCommitter = true;
+        }
+        if (sink.getCommittableSerializer().isPresent()) {
+            committer = true;
+        }
+
+        if (globalCommitter && committer && stateful) {
+            return new StatefulGlobalTwoPhaseCommittingSinkAdapter();
+        }
+        if (globalCommitter) {
+            return new GlobalCommittingSinkAdapter();
+        }
+        if (committer && stateful) {
+            return new StatefulTwoPhaseCommittingSinkAdapter();
+        }
+        if (committer) {
+            return new TwoPhaseCommittingSinkAdapter();
+        }
+        if (stateful) {
+            return new StatefulSinkAdapter();
+        }
+        return this;
+    }
+
+    private static class SinkWriterV1Adapter<InputT, CommT, WriterStateT>
+            implements StatefulSinkWriter<InputT, WriterStateT>,
+                    PrecommittingSinkWriter<InputT, CommT> {
+
+        private final org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                writer;
+        private boolean endOfInput = false;
+        private final WriterContextAdapter contextAdapter = new WriterContextAdapter();
+
+        public SinkWriterV1Adapter(
+                org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                        writer) {
+            this.writer = writer;
+        }
+
+        @Override
+        public void write(InputT element, Context context)
+                throws IOException, InterruptedException {
+            contextAdapter.setContext(context);
+            this.writer.write(element, contextAdapter);
+        }
+
+        @Override
+        public void flush(boolean endOfInput) throws IOException, InterruptedException {
+            this.endOfInput = endOfInput;
+        }
+
+        @Override
+        public List<WriterStateT> snapshotState(long checkpointId) throws IOException {
+            return writer.snapshotState(checkpointId);
+        }
+
+        @Override
+        public Collection<CommT> prepareCommit() throws IOException, InterruptedException {
+            return writer.prepareCommit(endOfInput);
+        }
+
+        @Override
+        public void close() throws Exception {
+            writer.close();
+        }
+
+        @Override
+        public void writeWatermark(Watermark watermark) throws IOException, InterruptedException {
+            writer.writeWatermark(watermark);
+        }
+    }
+
+    private static class WriterContextAdapter implements SinkWriter.Context {
+        private org.apache.flink.api.connector.sink2.SinkWriter.Context context;
+
+        public void setContext(org.apache.flink.api.connector.sink2.SinkWriter.Context context) {
+            this.context = context;
+        }
+
+        @Override
+        public long currentWatermark() {
+            return context.currentWatermark();
+        }
+
+        @Override
+        public Long timestamp() {
+            return context.timestamp();
+        }
+    }
+
+    private static class InitContextAdapter
+            implements org.apache.flink.api.connector.sink.Sink.InitContext {
+
+        private final InitContext context;
+
+        public InitContextAdapter(InitContext context) {
+            this.context = context;
+        }
+
+        @Override
+        public UserCodeClassLoader getUserCodeClassLoader() {
+            return context.getUserCodeClassLoader();
+        }
+
+        @Override
+        public MailboxExecutor getMailboxExecutor() {
+            return context.getMailboxExecutor();
+        }
+
+        @Override
+        public ProcessingTimeService getProcessingTimeService() {
+            return new ProcessingTimeServiceAdapter(context.getProcessingTimeService());
+        }
+
+        @Override
+        public int getSubtaskId() {
+            return context.getSubtaskId();
+        }
+
+        @Override
+        public int getNumberOfParallelSubtasks() {
+            return context.getNumberOfParallelSubtasks();
+        }
+
+        @Override
+        public SinkWriterMetricGroup metricGroup() {
+            return context.metricGroup();
+        }
+
+        @Override
+        public OptionalLong getRestoredCheckpointId() {
+            return context.getRestoredCheckpointId();
+        }
+
+        public InitializationContext asSerializationSchemaInitializationContext() {
+            return context.asSerializationSchemaInitializationContext();
+        }
+    }
+
+    private static class ProcessingTimeCallbackAdapter implements ProcessingTimeCallback {
+
+        private final ProcessingTimeService.ProcessingTimeCallback processingTimerCallback;
+
+        public ProcessingTimeCallbackAdapter(
+                ProcessingTimeService.ProcessingTimeCallback processingTimerCallback) {
+            this.processingTimerCallback = processingTimerCallback;
+        }
+
+        @Override
+        public void onProcessingTime(long time) throws IOException, InterruptedException {
+            processingTimerCallback.onProcessingTime(time);
+        }
+    }
+
+    private static class ProcessingTimeServiceAdapter implements ProcessingTimeService {
+
+        private final org.apache.flink.api.common.operators.ProcessingTimeService
+                processingTimeService;
+
+        public ProcessingTimeServiceAdapter(
+                org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService) {
+            this.processingTimeService = processingTimeService;
+        }
+
+        @Override
+        public long getCurrentProcessingTime() {
+            return processingTimeService.getCurrentProcessingTime();
+        }
+
+        @Override
+        public void registerProcessingTimer(
+                long time, ProcessingTimeCallback processingTimerCallback) {
+            processingTimeService.registerTimer(
+                    time, new ProcessingTimeCallbackAdapter(processingTimerCallback));
+        }
+    }
+
+    private static class CommitterAdapter<CommT> implements Committer<CommT> {
+
+        private final org.apache.flink.api.connector.sink.Committer<CommT> committer;
+
+        public CommitterAdapter(org.apache.flink.api.connector.sink.Committer<CommT> committer) {
+            this.committer = committer;
+        }
+
+        @Override
+        public void commit(Collection<CommitRequest<CommT>> commitRequests)
+                throws IOException, InterruptedException {
+            List<CommT> failed =
+                    committer.commit(
+                            commitRequests.stream()
+                                    .map(CommitRequest::getCommittable)
+                                    .collect(Collectors.toList()));
+            if (!failed.isEmpty()) {
+                Set<CommT> indexed = Collections.newSetFromMap(new IdentityHashMap<>());
+                indexed.addAll(failed);
+                commitRequests.stream()
+                        .filter(request -> indexed.contains(request.getCommittable()))
+                        .forEach(CommitRequest::retryLater);
+            }
+        }
+
+        @Override
+        public void close() throws Exception {
+            committer.close();
+        }
+    }
+
+    /** Main class to simulate SinkV1 with SinkV2. */
+    @VisibleForTesting
+    @Internal
+    public class PlainSinkAdapter implements Sink<InputT> {
+        @Override
+        public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+                throws IOException {
+            return SinkV1Adapter.this.createWriter(context);
+        }
+
+        @VisibleForTesting
+        public org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                getSink() {
+            return sink;
+        }
+    }
+
+    private class StatefulSinkAdapter extends PlainSinkAdapter
+            implements StatefulSink<InputT, WriterStateT> {
+        @Override
+        public StatefulSinkWriter<InputT, WriterStateT> restoreWriter(
+                InitContext context, Collection<WriterStateT> recoveredState) throws IOException {
+            org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                    sink.createWriter(
+                            new InitContextAdapter(context), new ArrayList<>(recoveredState));
+            return new SinkWriterV1Adapter<>(writer);
+        }
+
+        @Override
+        public SimpleVersionedSerializer<WriterStateT> getWriterStateSerializer() {
+            return sink.getWriterStateSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+    }
+
+    private class TwoPhaseCommittingSinkAdapter extends PlainSinkAdapter
+            implements TwoPhaseCommittingSink<InputT, CommT>, WithCompatibleState {
+        @Override
+        public Committer<CommT> createCommitter() throws IOException {
+            return new CommitterAdapter<>(
+                    sink.createCommitter().orElse(new SinkV1Adapter.NoopCommitter<>()));
+        }
+
+        @Override
+        public SimpleVersionedSerializer<CommT> getCommittableSerializer() {
+            return sink.getCommittableSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+
+        @Override
+        public Collection<String> getCompatibleWriterStateNames() {
+            return sink.getCompatibleStateNames();
+        }
+    }
+
+    private class GlobalCommittingSinkAdapter extends TwoPhaseCommittingSinkAdapter
+            implements WithPostCommitTopology<InputT, CommT> {
+
+        @Override
+        public void addPostCommitTopology(DataStream<CommittableMessage<CommT>> committables) {
+            StandardSinkTopologies.addGlobalCommitter(committables, GlobalCommitterAdapter::new);
+        }
+    }
+
+    private class StatefulTwoPhaseCommittingSinkAdapter extends StatefulSinkAdapter

Review comment:
       ISP(Interface Segregation Principle) is controlled by the interface not the implementation classes. It is fine to split the big old Sink interface based on ISP. Using one class to implement them all will not violate it. Because what the caller will see are fine-grained interfaces not the implementation class(es), especially when all of these classes are private inner classes and most logic is to wrap each other.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a826e83de723921dd06636571743486cdef432b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315) 
   * 3b172e6de001227e6e383b5e7431eed5b529f8be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a826e83de723921dd06636571743486cdef432b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315) 
   * 3b172e6de001227e6e383b5e7431eed5b529f8be UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795670801



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManager.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Manages the committables coming from one subtask. */
+class SubtaskCommittableManager<CommT> {
+    private CommittableSummary<CommT> summary;

Review comment:
       Good catch. I removed the committable summary from the `SubtaskCommittableManager` and only pass the necessary information now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794444172



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {

Review comment:
       I added a java doc




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795465944



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittableManagerImpl<CommT> implements CheckpointCommittableManager<CommT> {
+    /** Mapping of subtask id to {@link SubtaskCommittableManager}. */
+    private final Map<Integer, SubtaskCommittableManager<CommT>> subtasksCommittableManagers;
+
+    private final CommittableCollector<CommT> collector;

Review comment:
       I think I can just pass the subtaskId and numberOfTasks to the `CheckpointCommittableManager` and remove this way the reverse dependency.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795698138



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -19,250 +19,237 @@
 package org.apache.flink.streaming.runtime.translators;
 
 import org.apache.flink.annotation.Internal;
-import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
 import org.apache.flink.api.dag.Transformation;
-import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.graph.TransformationTranslator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
 import org.apache.flink.streaming.api.transformations.SinkTransformation;
+import org.apache.flink.streaming.api.transformations.StreamExchangeMode;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
-import org.apache.flink.streaming.runtime.operators.sink.SinkOperatorFactory;
-import org.apache.flink.streaming.util.graph.StreamGraphUtils;
-import org.apache.flink.util.FlinkRuntimeException;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.flink.streaming.runtime.operators.sink.WriterOperatorFactory;
+import org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner;
 
 import javax.annotation.Nullable;
 
-import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** A {@link TransformationTranslator} for the {@link SinkTransformation}. */
+import java.util.List;
+import java.util.Optional;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+
+/**
+ * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
+ * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ */
 @Internal
-public class SinkTransformationTranslator<InputT, CommT, WriterStateT, GlobalCommT>
-        implements TransformationTranslator<
-                Object, SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT>> {
+public class SinkTransformationTranslator<Input, Output>
+        implements TransformationTranslator<Output, SinkTransformation<Input, Output>> {
 
-    protected static final Logger LOG = LoggerFactory.getLogger(SinkTransformationTranslator.class);
+    private static final String COMMITTER_NAME = "Committer";
+    private static final String WRITER_NAME = "Writer";
 
     @Override
     public Collection<Integer> translateForBatch(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, true, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
-        return Collections.emptyList();
+            SinkTransformation<Input, Output> transformation, Context context) {
+        return translateForStreaming(transformation, context);
     }
 
     @Override
     public Collection<Integer> translateForStreaming(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> transformation,
-            Context context) {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), transformation);
-
-        final int parallelism = getParallelism(transformation, context);
-
-        try {
-            internalTranslate(transformation, parallelism, false, context);
-        } catch (IOException e) {
-            throw new FlinkRuntimeException(
-                    "Could not add the Committer or GlobalCommitter to the stream graph.", e);
-        }
+            SinkTransformation<Input, Output> transformation, Context context) {
 
+        SinkExpander<Input> expander =
+                new SinkExpander<>(
+                        transformation.getStream(),
+                        transformation.getSink(),
+                        transformation,
+                        context);
+        expander.expand();
         return Collections.emptyList();
     }
 
     /**
-     * Add the sink operators to the stream graph.
-     *
-     * @param sinkTransformation The sink transformation that committer and global committer belongs
-     *     to.
-     * @param writerParallelism The parallelism of the writer operator.
-     * @param batch Specifies if this sink is executed in batch mode.
+     * Expands the FLIP-143 Sink to a subtopology. Each part of the topology is created after the
+     * previous part of the topology has been completely configured by the user. For example, if a
+     * user explicitly sets the parallelism of the sink, each part of the subtopology can rely on
+     * the input having that parallelism.
      */
-    private void internalTranslate(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int writerParallelism,
-            boolean batch,
-            Context context)
-            throws IOException {
-
-        StreamGraphUtils.validateTransformationUid(context.getStreamGraph(), sinkTransformation);
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        boolean needsCommitterOperator =
-                batch && sink.getCommittableSerializer().isPresent()
-                        || sink.getGlobalCommittableSerializer().isPresent();
-        final int writerId =
-                addWriterAndCommitter(
-                        sinkTransformation,
-                        writerParallelism,
-                        batch,
-                        needsCommitterOperator,
-                        context);
-
-        if (needsCommitterOperator) {
-            addGlobalCommitter(writerId, sinkTransformation, batch, context);
+    private static class SinkExpander<T> {
+        private final SinkTransformation<T, ?> transformation;
+        private final Sink<T> sink;
+        private final Context context;
+        private final DataStream<T> inputStream;
+        private final StreamExecutionEnvironment executionEnvironment;
+        private boolean expanded;
+
+        public SinkExpander(
+                DataStream<T> inputStream,
+                Sink<T> sink,
+                SinkTransformation<T, ?> transformation,
+                Context context) {
+            this.inputStream = inputStream;
+            this.executionEnvironment = inputStream.getExecutionEnvironment();
+            this.transformation = transformation;
+            this.sink = sink;
+            this.context = context;
         }
-    }
-
-    /**
-     * Add a sink writer node to the stream graph.
-     *
-     * @param sinkTransformation The transformation that the writer belongs to
-     * @param parallelism The parallelism of the writer
-     * @param batch Specifies if this sink is executed in batch mode.
-     * @param shouldEmit Specifies whether the write should emit committables.
-     * @return The stream node id of the writer
-     */
-    private int addWriterAndCommitter(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            int parallelism,
-            boolean batch,
-            boolean shouldEmit,
-            Context context) {
 
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-        checkState(sinkTransformation.getInputs().size() == 1);
-        @SuppressWarnings("unchecked")
-        final Transformation<InputT> input =
-                (Transformation<InputT>) sinkTransformation.getInputs().get(0);
-        final TypeInformation<InputT> inputTypeInfo = input.getOutputType();
-
-        final StreamOperatorFactory<byte[]> factory =
-                new SinkOperatorFactory<>(sink, batch, shouldEmit);
-
-        final ChainingStrategy chainingStrategy = sinkTransformation.getChainingStrategy();
-
-        if (chainingStrategy != null) {
-            factory.setChainingStrategy(chainingStrategy);
+        private void expand() {
+            final int sizeBefore = executionEnvironment.getTransformations().size();
+            if (expanded) {
+                // may be called twice for multi-staged application, make sure to expand only once
+                return;
+            }
+            expanded = true;
+
+            DataStream<T> prewritten = inputStream;
+            if (sink instanceof WithPreWriteTopology) {
+                prewritten =
+                        adjustTransformations(
+                                prewritten, ((WithPreWriteTopology<T>) sink)::addPreWriteTopology);
+            }
+
+            if (sink instanceof TwoPhaseCommittingSink) {
+                addCommittingTopology(sink, prewritten);
+            } else {
+                adjustTransformations(
+                        prewritten,
+                        input ->
+                                input.transform(
+                                        WRITER_NAME,
+                                        CommittableMessageTypeInfo.noOutput(),
+                                        new WriterOperatorFactory<>(sink)));
+            }
+            final List<Transformation<?>> sinkTransformations =
+                    executionEnvironment
+                            .getTransformations()
+                            .subList(sizeBefore, executionEnvironment.getTransformations().size());
+            sinkTransformations.forEach(context::transform);
         }
 
-        final String format = batch && shouldEmit ? "Sink %s Writer" : "Sink %s";
-
-        return addOperatorToStreamGraph(
-                factory,
-                context.getStreamNodeIds(input),
-                inputTypeInfo,
-                TypeInformation.of(byte[].class),
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid(),
-                parallelism,
-                sinkTransformation.getMaxParallelism(),
-                sinkTransformation,
-                context);
-    }
-
-    /**
-     * Try to add a sink global committer to the stream graph.
-     *
-     * @param inputId The global committer's input stream node id.
-     * @param sinkTransformation The transformation that the global committer belongs to.
-     * @param batch Specifies if this sink is executed in batch mode.
-     */
-    private void addGlobalCommitter(
-            int inputId,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            boolean batch,
-            Context context) {
-
-        Sink<InputT, CommT, WriterStateT, GlobalCommT> sink = sinkTransformation.getSink();
-
-        final String format = batch ? "Sink %s Committer" : "Sink %s Global Committer";
-
-        addOperatorToStreamGraph(
-                new CommitterOperatorFactory<>(sink, batch),
-                Collections.singletonList(inputId),
-                TypeInformation.of(byte[].class),
-                null,
-                String.format(format, sinkTransformation.getName()),
-                sinkTransformation.getUid() == null
-                        ? null
-                        : String.format(format, sinkTransformation.getUid()),
-                1,
-                1,
-                sinkTransformation,
-                context);
-    }
-
-    private int getParallelism(
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        return sinkTransformation.getParallelism() != ExecutionConfig.PARALLELISM_DEFAULT
-                ? sinkTransformation.getParallelism()
-                : context.getStreamGraph().getExecutionConfig().getParallelism();
-    }
-
-    /**
-     * Add a operator to the {@link StreamGraph}.
-     *
-     * @param operatorFactory The operator factory
-     * @param inputs A collection of upstream stream node ids.
-     * @param inTypeInfo The input type information of the operator
-     * @param outTypInfo The output type information of the operator
-     * @param name The name of the operator.
-     * @param uid The uid of the operator.
-     * @param parallelism The parallelism of the operator
-     * @param maxParallelism The max parallelism of the operator
-     * @param sinkTransformation The sink transformation which the operator belongs to
-     * @return The stream node id of the operator
-     */
-    private <IN, OUT> int addOperatorToStreamGraph(
-            StreamOperatorFactory<OUT> operatorFactory,
-            Collection<Integer> inputs,
-            TypeInformation<IN> inTypeInfo,
-            TypeInformation<OUT> outTypInfo,
-            String name,
-            @Nullable String uid,
-            int parallelism,
-            int maxParallelism,
-            SinkTransformation<InputT, CommT, WriterStateT, GlobalCommT> sinkTransformation,
-            Context context) {
-        final StreamGraph streamGraph = context.getStreamGraph();
-        final String slotSharingGroup = context.getSlotSharingGroup();
-        final int transformationId = Transformation.getNewNodeId();
-
-        streamGraph.addOperator(
-                transformationId,
-                slotSharingGroup,
-                sinkTransformation.getCoLocationGroupKey(),
-                operatorFactory,
-                inTypeInfo,
-                outTypInfo,
-                name);
+        private <T, CommT> void addCommittingTopology(Sink<T> sink, DataStream<T> inputStream) {
+            TwoPhaseCommittingSink<T, CommT> committingSink =
+                    (TwoPhaseCommittingSink<T, CommT>) sink;
+            TypeInformation<CommittableMessage<CommT>> typeInformation =
+                    CommittableMessageTypeInfo.of(committingSink::getCommittableSerializer);
+
+            DataStream<CommittableMessage<CommT>> written =
+                    adjustTransformations(
+                            inputStream,
+                            input ->
+                                    input.transform(
+                                            WRITER_NAME,
+                                            typeInformation,
+                                            new WriterOperatorFactory<>(sink)));
+
+            DataStream<CommittableMessage<CommT>> precommitted = addFailOverRegion(written);
+
+            if (sink instanceof WithPreCommitTopology) {
+                precommitted =
+                        adjustTransformations(
+                                precommitted,
+                                ((WithPreCommitTopology<T, CommT>) sink)::addPreCommitTopology);
+            }
+            DataStream<CommittableMessage<CommT>> committed =
+                    adjustTransformations(
+                            precommitted,
+                            pc ->
+                                    pc.transform(
+                                            COMMITTER_NAME,
+                                            typeInformation,
+                                            new CommitterOperatorFactory<>(committingSink)));
+            if (sink instanceof WithPostCommitTopology) {
+                DataStream<CommittableMessage<CommT>> postcommitted = addFailOverRegion(committed);
+                adjustTransformations(
+                        postcommitted,
+                        pc -> {
+                            ((WithPostCommitTopology<T, CommT>) sink).addPostCommitTopology(pc);
+                            return null;
+                        });
+            }
+        }
 
-        streamGraph.setParallelism(transformationId, parallelism);
-        streamGraph.setMaxParallelism(transformationId, maxParallelism);
+        /**
+         * Adds a batch exchange that materializes the output first. This is a no-op in STREAMING.
+         */
+        private <T> DataStream<T> addFailOverRegion(DataStream<T> input) {
+            return new DataStream<>(
+                    executionEnvironment,
+                    new PartitionTransformation<>(
+                            input.getTransformation(),
+                            new ForwardPartitioner<>(),
+                            StreamExchangeMode.BATCH));
+        }
 
-        StreamGraphUtils.configureBufferTimeout(
-                streamGraph,
-                transformationId,
-                sinkTransformation,
-                context.getDefaultBufferTimeout());
-        if (uid != null) {
-            streamGraph.setTransformationUID(transformationId, uid);
+        private <T, R> R adjustTransformations(
+                DataStream<T> inputStream, Function<DataStream<T>, R> action) {
+            int numTransformsBefore = executionEnvironment.getTransformations().size();
+            R result = action.apply(inputStream);
+            List<Transformation<?>> transformations = executionEnvironment.getTransformations();
+            List<Transformation<?>> expandedTransformations =
+                    transformations.subList(numTransformsBefore, transformations.size());
+            for (Transformation<?> subTransformation : expandedTransformations) {
+                concatUid(
+                        subTransformation,
+                        Transformation::getUid,
+                        Transformation::setUid,
+                        subTransformation.getName());
+                concatProperty(subTransformation, Transformation::getName, Transformation::setName);
+                concatProperty(
+                        subTransformation,
+                        Transformation::getDescription,
+                        Transformation::setDescription);
+
+                Optional<SlotSharingGroup> ssg = transformation.getSlotSharingGroup();
+                if (ssg.isPresent() && !subTransformation.getSlotSharingGroup().isPresent()) {
+                    subTransformation.setSlotSharingGroup(ssg.get());
+                }
+                subTransformation.setParallelism(transformation.getParallelism());

Review comment:
       I had another look and it might be easier than we initially thought. By default, all transformations have specific default parallelism [1]. So I can simply do the following 
   
   ```java
                   if (subTransformation.getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT) {
                       subTransformation.setParallelism(transformation.getParallelism());
                   }
   
   ```
   As soon as users set the parallelism, it won't change during the translation. Another important point is that the `sink transformation` parallelism is equal to the `env default` parallelism. [2]
   
   [1] https://github.com/apache/flink/blob/cd8ea8d5b207569f68acc5a3c8db95cd2ca47ba6/flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java#L87
   [2] https://github.com/fapaul/flink/blob/e135c4f6208fb5efe5c0c0640f90d82f588e244d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java#L73




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   * b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791937510



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {
+    private CommittableSummary<CommT> summary;
+    private final Deque<CommitRequestImpl<CommT>> requests;
+    private int numDrained;
+    private int numFailed;
+
+    SubtaskCommittables(CommittableSummary<CommT> summary) {
+        this(summary, Collections.emptyList(), 0, 0);
+    }
+
+    SubtaskCommittables(
+            CommittableSummary<CommT> summary,
+            Collection<CommitRequestImpl<CommT>> requests,
+            int numDrained,
+            int numFailed) {
+        this.summary = checkNotNull(summary);
+        this.requests = new ArrayDeque<>(checkNotNull(requests));
+        this.numDrained = numDrained;
+        this.numFailed = numFailed;
+    }
+
+    void add(CommittableWithLineage<CommT> committable) {
+        add(committable.getCommittable());
+    }
+
+    void add(CommT committable) {
+        requests.add(new CommitRequestImpl<>(committable));
+    }
+
+    boolean hasReceivedAll() {
+        return getNumCommittables() == summary.getNumberOfCommittables();
+    }
+
+    int getNumCommittables() {
+        return requests.size() + numDrained + numFailed;
+    }
+
+    int getNumPending() {
+        return summary.getNumberOfCommittables() - (numDrained + numFailed);
+    }
+
+    int getNumFailed() {
+        return numFailed;
+    }
+
+    boolean isFinished() {
+        return getNumPending() == 0;
+    }
+
+    Stream<CommitRequestImpl<CommT>> getPendingRequests() {
+        return requests.stream().filter(c -> !c.isFinished());
+    }
+
+    List<CommittableWithLineage<CommT>> drainCommitted() {

Review comment:
       I think java doc is required for this method. Will this method be called multiple times? How to make the numbers being synched with the `summary`, i.e. `hasReceivedAll()`? 
   
   There many numbers defined in this class that need more detailed description about the relationship between them:
   1. total number of commitables in `summary`
   2. total number of `CommitRequestImpl` in `request`
      2.1 number of unfinished request
      2.2 number of failed request
      2.3 number of request that will be drained and removed
   3. numFailed, accumulated if this method would be called multiple time.
   4. numDrained, accumulated if this method would be called multiple time.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r792946618



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.serialization.SerializationSchema.InitializationContext;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.Sink.ProcessingTimeService;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.StatefulSink;
+import org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter;
+import org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/** Translates Sink V1 into Sink V2. */
+@Internal
+public class SinkV1Adapter<InputT, CommT, WriterStateT, GlobalCommT> implements Sink<InputT> {
+
+    private final org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+            sink;
+
+    private SinkV1Adapter(
+            org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                    sink) {
+        this.sink = sink;
+    }
+
+    public static <InputT> Sink<InputT> wrap(
+            org.apache.flink.api.connector.sink.Sink<InputT, ?, ?, ?> sink) {
+        return new SinkV1Adapter<>(sink).asSpecializedSink();
+    }
+
+    @Override
+    public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+            throws IOException {
+        org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                sink.createWriter(new InitContextAdapter(context), Collections.emptyList());
+        return new SinkWriterV1Adapter<>(writer);
+    }
+
+    public Sink<InputT> asSpecializedSink() {
+        boolean stateful = false;
+        boolean globalCommitter = false;
+        boolean committer = false;
+        if (sink.getWriterStateSerializer().isPresent()) {
+            stateful = true;
+        }
+        if (sink.getGlobalCommittableSerializer().isPresent()) {
+            globalCommitter = true;
+        }
+        if (sink.getCommittableSerializer().isPresent()) {
+            committer = true;
+        }
+
+        if (globalCommitter && committer && stateful) {
+            return new StatefulGlobalTwoPhaseCommittingSinkAdapter();
+        }
+        if (globalCommitter) {
+            return new GlobalCommittingSinkAdapter();
+        }
+        if (committer && stateful) {
+            return new StatefulTwoPhaseCommittingSinkAdapter();
+        }
+        if (committer) {
+            return new TwoPhaseCommittingSinkAdapter();
+        }
+        if (stateful) {
+            return new StatefulSinkAdapter();
+        }
+        return this;
+    }
+
+    private static class SinkWriterV1Adapter<InputT, CommT, WriterStateT>
+            implements StatefulSinkWriter<InputT, WriterStateT>,
+                    PrecommittingSinkWriter<InputT, CommT> {
+
+        private final org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                writer;
+        private boolean endOfInput = false;
+        private final WriterContextAdapter contextAdapter = new WriterContextAdapter();
+
+        public SinkWriterV1Adapter(
+                org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                        writer) {
+            this.writer = writer;
+        }
+
+        @Override
+        public void write(InputT element, Context context)
+                throws IOException, InterruptedException {
+            contextAdapter.setContext(context);
+            this.writer.write(element, contextAdapter);
+        }
+
+        @Override
+        public void flush(boolean endOfInput) throws IOException, InterruptedException {
+            this.endOfInput = endOfInput;
+        }
+
+        @Override
+        public List<WriterStateT> snapshotState(long checkpointId) throws IOException {
+            return writer.snapshotState(checkpointId);
+        }
+
+        @Override
+        public Collection<CommT> prepareCommit() throws IOException, InterruptedException {
+            return writer.prepareCommit(endOfInput);
+        }
+
+        @Override
+        public void close() throws Exception {
+            writer.close();
+        }
+
+        @Override
+        public void writeWatermark(Watermark watermark) throws IOException, InterruptedException {
+            writer.writeWatermark(watermark);
+        }
+    }
+
+    private static class WriterContextAdapter implements SinkWriter.Context {
+        private org.apache.flink.api.connector.sink2.SinkWriter.Context context;
+
+        public void setContext(org.apache.flink.api.connector.sink2.SinkWriter.Context context) {
+            this.context = context;
+        }
+
+        @Override
+        public long currentWatermark() {
+            return context.currentWatermark();
+        }
+
+        @Override
+        public Long timestamp() {
+            return context.timestamp();
+        }
+    }
+
+    private static class InitContextAdapter
+            implements org.apache.flink.api.connector.sink.Sink.InitContext {
+
+        private final InitContext context;
+
+        public InitContextAdapter(InitContext context) {
+            this.context = context;
+        }
+
+        @Override
+        public UserCodeClassLoader getUserCodeClassLoader() {
+            return context.getUserCodeClassLoader();
+        }
+
+        @Override
+        public MailboxExecutor getMailboxExecutor() {
+            return context.getMailboxExecutor();
+        }
+
+        @Override
+        public ProcessingTimeService getProcessingTimeService() {
+            return new ProcessingTimeServiceAdapter(context.getProcessingTimeService());
+        }
+
+        @Override
+        public int getSubtaskId() {
+            return context.getSubtaskId();
+        }
+
+        @Override
+        public int getNumberOfParallelSubtasks() {
+            return context.getNumberOfParallelSubtasks();
+        }
+
+        @Override
+        public SinkWriterMetricGroup metricGroup() {
+            return context.metricGroup();
+        }
+
+        @Override
+        public OptionalLong getRestoredCheckpointId() {
+            return context.getRestoredCheckpointId();
+        }
+
+        public InitializationContext asSerializationSchemaInitializationContext() {
+            return context.asSerializationSchemaInitializationContext();
+        }
+    }
+
+    private static class ProcessingTimeCallbackAdapter implements ProcessingTimeCallback {
+
+        private final ProcessingTimeService.ProcessingTimeCallback processingTimerCallback;
+
+        public ProcessingTimeCallbackAdapter(
+                ProcessingTimeService.ProcessingTimeCallback processingTimerCallback) {
+            this.processingTimerCallback = processingTimerCallback;
+        }
+
+        @Override
+        public void onProcessingTime(long time) throws IOException, InterruptedException {
+            processingTimerCallback.onProcessingTime(time);
+        }
+    }
+
+    private static class ProcessingTimeServiceAdapter implements ProcessingTimeService {
+
+        private final org.apache.flink.api.common.operators.ProcessingTimeService
+                processingTimeService;
+
+        public ProcessingTimeServiceAdapter(
+                org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService) {
+            this.processingTimeService = processingTimeService;
+        }
+
+        @Override
+        public long getCurrentProcessingTime() {
+            return processingTimeService.getCurrentProcessingTime();
+        }
+
+        @Override
+        public void registerProcessingTimer(
+                long time, ProcessingTimeCallback processingTimerCallback) {
+            processingTimeService.registerTimer(
+                    time, new ProcessingTimeCallbackAdapter(processingTimerCallback));
+        }
+    }
+
+    private static class CommitterAdapter<CommT> implements Committer<CommT> {
+
+        private final org.apache.flink.api.connector.sink.Committer<CommT> committer;
+
+        public CommitterAdapter(org.apache.flink.api.connector.sink.Committer<CommT> committer) {
+            this.committer = committer;
+        }
+
+        @Override
+        public void commit(Collection<CommitRequest<CommT>> commitRequests)
+                throws IOException, InterruptedException {
+            List<CommT> failed =
+                    committer.commit(
+                            commitRequests.stream()
+                                    .map(CommitRequest::getCommittable)
+                                    .collect(Collectors.toList()));
+            if (!failed.isEmpty()) {
+                Set<CommT> indexed = Collections.newSetFromMap(new IdentityHashMap<>());
+                indexed.addAll(failed);
+                commitRequests.stream()
+                        .filter(request -> indexed.contains(request.getCommittable()))
+                        .forEach(CommitRequest::retryLater);
+            }
+        }
+
+        @Override
+        public void close() throws Exception {
+            committer.close();
+        }
+    }
+
+    /** Main class to simulate SinkV1 with SinkV2. */
+    @VisibleForTesting
+    @Internal
+    public class PlainSinkAdapter implements Sink<InputT> {
+        @Override
+        public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+                throws IOException {
+            return SinkV1Adapter.this.createWriter(context);
+        }
+
+        @VisibleForTesting

Review comment:
       redundant? There is one at class level.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java
##########
@@ -40,32 +45,46 @@
 
     private final PhysicalTransformation<T> transformation;
 
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, StreamSink<T> operator) {
-        this.transformation =
-                (PhysicalTransformation<T>)
-                        new LegacySinkTransformation<>(
-                                inputStream.getTransformation(),
-                                "Unnamed",
-                                operator,
-                                inputStream.getExecutionEnvironment().getParallelism());
-    }
-
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, Sink<T, ?, ?, ?> sink) {
-        transformation =
-                (PhysicalTransformation<T>)
-                        new SinkTransformation<>(
-                                inputStream.getTransformation(),
-                                sink,
-                                "Unnamed",
-                                inputStream.getExecutionEnvironment().getParallelism());
+    protected DataStreamSink(PhysicalTransformation<T> transformation) {
+        this.transformation = checkNotNull(transformation);
+    }
+
+    static <T> DataStreamSink<T> forSinkFunction(
+            DataStream<T> inputStream, SinkFunction<T> sinkFunction) {

Review comment:
       Why replace the operator with the sinkFunction, which is planned to deprecate? Which means that the modified code will be changed again after two releases when the sinkFunction has to be deleted. Will the `StreamSink` be deprecated together with `SinkFunction`?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/WriterOperator.java
##########
@@ -56,105 +59,64 @@
  * same parallelism or send them downstream to a {@link CommitterOperator} with a different
  * parallelism.
  *
- * <p>The operator may be part of a sink pipeline and is the first operator. There are currently two
- * ways this operator is used:
- *
- * <ul>
- *   <li>In streaming mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
+ * <p>The operator may be part of a sink pipeline and is the first operator.

Review comment:
       I think it still make sense to add more info about relationship between these operators, i.e. WriterOperator and CommitterOperator and the SinkWriter, Committer, GlobalCommitter.
   
   "may be" - does it mean there could be a sink pipeline without a `WriterOperator` ? Could we describe such scenario a little bit here to let the user understand it better.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/PartitionTransformationTranslator.java
##########
@@ -70,13 +73,15 @@
 
         List<Integer> resultIds = new ArrayList<>();
 
+        StreamExchangeMode exchangeMode = transformation.getExchangeMode();
+        if (!supportsBatchExchange && exchangeMode == StreamExchangeMode.BATCH) {
+            exchangeMode = StreamExchangeMode.UNDEFINED;

Review comment:
       hmm, It would be better to add some description to explain:
   1. why `translateForStreamingInternal(...)` is possible to be called with a transformation whose exchangeMode is `StreamExchangeMode.BATCH`?
   2. after this modification, a BATCH exchangeMode transformation could become a PIPELINED. Will there be any potential issue?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java
##########
@@ -40,32 +45,46 @@
 
     private final PhysicalTransformation<T> transformation;
 
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, StreamSink<T> operator) {
-        this.transformation =
-                (PhysicalTransformation<T>)
-                        new LegacySinkTransformation<>(
-                                inputStream.getTransformation(),
-                                "Unnamed",
-                                operator,
-                                inputStream.getExecutionEnvironment().getParallelism());
-    }
-
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, Sink<T, ?, ?, ?> sink) {
-        transformation =
-                (PhysicalTransformation<T>)
-                        new SinkTransformation<>(
-                                inputStream.getTransformation(),
-                                sink,
-                                "Unnamed",
-                                inputStream.getExecutionEnvironment().getParallelism());
+    protected DataStreamSink(PhysicalTransformation<T> transformation) {
+        this.transformation = checkNotNull(transformation);
+    }
+
+    static <T> DataStreamSink<T> forSinkFunction(
+            DataStream<T> inputStream, SinkFunction<T> sinkFunction) {
+        StreamSink<T> sinkOperator = new StreamSink<>(sinkFunction);
+        PhysicalTransformation<T> transformation =
+                new LegacySinkTransformation<>(
+                        inputStream.getTransformation(),
+                        "Unnamed",
+                        sinkOperator,
+                        inputStream.getExecutionEnvironment().getParallelism());
+        inputStream.getExecutionEnvironment().addOperator(transformation);
+        return new DataStreamSink<>(transformation);
+    }
+
+    static <T> DataStreamSink<T> forSink(DataStream<T> inputStream, Sink<T> sink) {
+        StreamExecutionEnvironment executionEnvironment = inputStream.getExecutionEnvironment();
+        SinkTransformation<T, T> transformation =
+                new SinkTransformation<>(
+                        inputStream,
+                        sink,
+                        inputStream.getTransformation(),
+                        inputStream.getType(),
+                        "Sink",
+                        executionEnvironment.getParallelism());
         inputStream.getExecutionEnvironment().addOperator(transformation);
+        return new DataStreamSink<>(transformation);
+    }
+
+    @Internal
+    public static <T> DataStreamSink<T> forSinkV1(
+            DataStream<T> inputStream, org.apache.flink.api.connector.sink.Sink<T, ?, ?, ?> sink) {
+        return forSink(inputStream, SinkV1Adapter.wrap(sink));
     }
 
     /** Returns the transformation that contains the actual sink operator of this sink. */
     @Internal
-    public Transformation<T> getTransformation() {
+    public Transformation<?> getTransformation() {

Review comment:
       Java doc for reducing the type restriction

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterStateHandler.java
##########
@@ -17,34 +17,27 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.runtime.state.StateInitializationContext;
-import org.apache.flink.util.function.FunctionWithException;
-
-import java.io.Serializable;
-import java.util.List;
 
 /**
  * Manages the state of a {@link org.apache.flink.api.connector.sink.SinkWriter}. There are only two
  * flavors: stateless handled by {@link StatelessSinkWriterStateHandler} and stateful handled with
  * {@link StatefulSinkWriterStateHandler}.
  *
- * @param <WriterStateT>
+ * @param <InputT> the input type
  */
-interface SinkWriterStateHandler<WriterStateT> extends Serializable {
-    /**
-     * Extracts the writer state from the {@link StateInitializationContext}. The state will be used
-     * to create the writer.
-     */
-    List<WriterStateT> initializeState(StateInitializationContext context) throws Exception;
+interface SinkWriterStateHandler<InputT> {
 
     /**
-     * Stores the state of the supplier. The supplier should only be queried once.
+     * Stores the state of the writer.
      *
-     * @param stateExtractor
-     * @param checkpointId
+     * @param checkpointId the checkpointId
      */
-    void snapshotState(
-            FunctionWithException<Long, List<WriterStateT>, Exception> stateExtractor,
-            long checkpointId)
+    void snapshotState(long checkpointId) throws Exception;
+
+    /** Creates a writer, potentially using state from {@link StateInitializationContext}. */
+    SinkWriter<InputT> createWriter(InitContext initContext, StateInitializationContext context)

Review comment:
       This is the hiding magic method that neither the class name nor the java doc has described. I would suggest using a more feasible class name to cover this functionality.
   

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/WriterOperator.java
##########
@@ -56,105 +59,64 @@
  * same parallelism or send them downstream to a {@link CommitterOperator} with a different
  * parallelism.
  *
- * <p>The operator may be part of a sink pipeline and is the first operator. There are currently two
- * ways this operator is used:
- *
- * <ul>
- *   <li>In streaming mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
+ * <p>The operator may be part of a sink pipeline and is the first operator.
  *
  * @param <InputT> the type of the committable
  * @param <CommT> the type of the committable (to send to downstream operators)
- * @param <WriterStateT> the type of the writer state for stateful sinks
  */
-class SinkOperator<InputT, CommT, WriterStateT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<InputT, byte[]>, BoundedOneInput {
+class WriterOperator<InputT, CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>

Review comment:
       might `SinkWriterOperator` be the better naming choice? Using `WriterOperator` to replace `SinkOperator` is a big change from the domain design perspective. We should then explain what is writer, what is sink, what is the relationship between them etc. Another fact is that both `Sink` and `SinkWriter` are used internally.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.serialization.SerializationSchema.InitializationContext;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.Sink.ProcessingTimeService;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.StatefulSink;
+import org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter;
+import org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/** Translates Sink V1 into Sink V2. */
+@Internal
+public class SinkV1Adapter<InputT, CommT, WriterStateT, GlobalCommT> implements Sink<InputT> {
+
+    private final org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+            sink;
+
+    private SinkV1Adapter(
+            org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                    sink) {
+        this.sink = sink;
+    }
+
+    public static <InputT> Sink<InputT> wrap(
+            org.apache.flink.api.connector.sink.Sink<InputT, ?, ?, ?> sink) {
+        return new SinkV1Adapter<>(sink).asSpecializedSink();
+    }
+
+    @Override
+    public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+            throws IOException {
+        org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                sink.createWriter(new InitContextAdapter(context), Collections.emptyList());
+        return new SinkWriterV1Adapter<>(writer);
+    }
+
+    public Sink<InputT> asSpecializedSink() {
+        boolean stateful = false;
+        boolean globalCommitter = false;
+        boolean committer = false;
+        if (sink.getWriterStateSerializer().isPresent()) {
+            stateful = true;
+        }
+        if (sink.getGlobalCommittableSerializer().isPresent()) {
+            globalCommitter = true;
+        }
+        if (sink.getCommittableSerializer().isPresent()) {
+            committer = true;
+        }
+
+        if (globalCommitter && committer && stateful) {
+            return new StatefulGlobalTwoPhaseCommittingSinkAdapter();
+        }
+        if (globalCommitter) {
+            return new GlobalCommittingSinkAdapter();
+        }
+        if (committer && stateful) {
+            return new StatefulTwoPhaseCommittingSinkAdapter();
+        }
+        if (committer) {
+            return new TwoPhaseCommittingSinkAdapter();
+        }
+        if (stateful) {
+            return new StatefulSinkAdapter();
+        }
+        return this;
+    }
+
+    private static class SinkWriterV1Adapter<InputT, CommT, WriterStateT>
+            implements StatefulSinkWriter<InputT, WriterStateT>,
+                    PrecommittingSinkWriter<InputT, CommT> {
+
+        private final org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                writer;
+        private boolean endOfInput = false;
+        private final WriterContextAdapter contextAdapter = new WriterContextAdapter();
+
+        public SinkWriterV1Adapter(
+                org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT>
+                        writer) {
+            this.writer = writer;
+        }
+
+        @Override
+        public void write(InputT element, Context context)
+                throws IOException, InterruptedException {
+            contextAdapter.setContext(context);
+            this.writer.write(element, contextAdapter);
+        }
+
+        @Override
+        public void flush(boolean endOfInput) throws IOException, InterruptedException {
+            this.endOfInput = endOfInput;
+        }
+
+        @Override
+        public List<WriterStateT> snapshotState(long checkpointId) throws IOException {
+            return writer.snapshotState(checkpointId);
+        }
+
+        @Override
+        public Collection<CommT> prepareCommit() throws IOException, InterruptedException {
+            return writer.prepareCommit(endOfInput);
+        }
+
+        @Override
+        public void close() throws Exception {
+            writer.close();
+        }
+
+        @Override
+        public void writeWatermark(Watermark watermark) throws IOException, InterruptedException {
+            writer.writeWatermark(watermark);
+        }
+    }
+
+    private static class WriterContextAdapter implements SinkWriter.Context {
+        private org.apache.flink.api.connector.sink2.SinkWriter.Context context;
+
+        public void setContext(org.apache.flink.api.connector.sink2.SinkWriter.Context context) {
+            this.context = context;
+        }
+
+        @Override
+        public long currentWatermark() {
+            return context.currentWatermark();
+        }
+
+        @Override
+        public Long timestamp() {
+            return context.timestamp();
+        }
+    }
+
+    private static class InitContextAdapter
+            implements org.apache.flink.api.connector.sink.Sink.InitContext {
+
+        private final InitContext context;
+
+        public InitContextAdapter(InitContext context) {
+            this.context = context;
+        }
+
+        @Override
+        public UserCodeClassLoader getUserCodeClassLoader() {
+            return context.getUserCodeClassLoader();
+        }
+
+        @Override
+        public MailboxExecutor getMailboxExecutor() {
+            return context.getMailboxExecutor();
+        }
+
+        @Override
+        public ProcessingTimeService getProcessingTimeService() {
+            return new ProcessingTimeServiceAdapter(context.getProcessingTimeService());
+        }
+
+        @Override
+        public int getSubtaskId() {
+            return context.getSubtaskId();
+        }
+
+        @Override
+        public int getNumberOfParallelSubtasks() {
+            return context.getNumberOfParallelSubtasks();
+        }
+
+        @Override
+        public SinkWriterMetricGroup metricGroup() {
+            return context.metricGroup();
+        }
+
+        @Override
+        public OptionalLong getRestoredCheckpointId() {
+            return context.getRestoredCheckpointId();
+        }
+
+        public InitializationContext asSerializationSchemaInitializationContext() {
+            return context.asSerializationSchemaInitializationContext();
+        }
+    }
+
+    private static class ProcessingTimeCallbackAdapter implements ProcessingTimeCallback {
+
+        private final ProcessingTimeService.ProcessingTimeCallback processingTimerCallback;
+
+        public ProcessingTimeCallbackAdapter(
+                ProcessingTimeService.ProcessingTimeCallback processingTimerCallback) {
+            this.processingTimerCallback = processingTimerCallback;
+        }
+
+        @Override
+        public void onProcessingTime(long time) throws IOException, InterruptedException {
+            processingTimerCallback.onProcessingTime(time);
+        }
+    }
+
+    private static class ProcessingTimeServiceAdapter implements ProcessingTimeService {
+
+        private final org.apache.flink.api.common.operators.ProcessingTimeService
+                processingTimeService;
+
+        public ProcessingTimeServiceAdapter(
+                org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService) {
+            this.processingTimeService = processingTimeService;
+        }
+
+        @Override
+        public long getCurrentProcessingTime() {
+            return processingTimeService.getCurrentProcessingTime();
+        }
+
+        @Override
+        public void registerProcessingTimer(
+                long time, ProcessingTimeCallback processingTimerCallback) {
+            processingTimeService.registerTimer(
+                    time, new ProcessingTimeCallbackAdapter(processingTimerCallback));
+        }
+    }
+
+    private static class CommitterAdapter<CommT> implements Committer<CommT> {
+
+        private final org.apache.flink.api.connector.sink.Committer<CommT> committer;
+
+        public CommitterAdapter(org.apache.flink.api.connector.sink.Committer<CommT> committer) {
+            this.committer = committer;
+        }
+
+        @Override
+        public void commit(Collection<CommitRequest<CommT>> commitRequests)
+                throws IOException, InterruptedException {
+            List<CommT> failed =
+                    committer.commit(
+                            commitRequests.stream()
+                                    .map(CommitRequest::getCommittable)
+                                    .collect(Collectors.toList()));
+            if (!failed.isEmpty()) {
+                Set<CommT> indexed = Collections.newSetFromMap(new IdentityHashMap<>());
+                indexed.addAll(failed);
+                commitRequests.stream()
+                        .filter(request -> indexed.contains(request.getCommittable()))
+                        .forEach(CommitRequest::retryLater);
+            }
+        }
+
+        @Override
+        public void close() throws Exception {
+            committer.close();
+        }
+    }
+
+    /** Main class to simulate SinkV1 with SinkV2. */
+    @VisibleForTesting
+    @Internal
+    public class PlainSinkAdapter implements Sink<InputT> {
+        @Override
+        public SinkWriterV1Adapter<InputT, CommT, WriterStateT> createWriter(InitContext context)
+                throws IOException {
+            return SinkV1Adapter.this.createWriter(context);
+        }
+
+        @VisibleForTesting
+        public org.apache.flink.api.connector.sink.Sink<InputT, CommT, WriterStateT, GlobalCommT>
+                getSink() {
+            return sink;
+        }
+    }
+
+    private class StatefulSinkAdapter extends PlainSinkAdapter
+            implements StatefulSink<InputT, WriterStateT> {
+        @Override
+        public StatefulSinkWriter<InputT, WriterStateT> restoreWriter(
+                InitContext context, Collection<WriterStateT> recoveredState) throws IOException {
+            org.apache.flink.api.connector.sink.SinkWriter<InputT, CommT, WriterStateT> writer =
+                    sink.createWriter(
+                            new InitContextAdapter(context), new ArrayList<>(recoveredState));
+            return new SinkWriterV1Adapter<>(writer);
+        }
+
+        @Override
+        public SimpleVersionedSerializer<WriterStateT> getWriterStateSerializer() {
+            return sink.getWriterStateSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+    }
+
+    private class TwoPhaseCommittingSinkAdapter extends PlainSinkAdapter
+            implements TwoPhaseCommittingSink<InputT, CommT>, WithCompatibleState {
+        @Override
+        public Committer<CommT> createCommitter() throws IOException {
+            return new CommitterAdapter<>(
+                    sink.createCommitter().orElse(new SinkV1Adapter.NoopCommitter<>()));
+        }
+
+        @Override
+        public SimpleVersionedSerializer<CommT> getCommittableSerializer() {
+            return sink.getCommittableSerializer()
+                    .orElseThrow(
+                            () ->
+                                    new IllegalStateException(
+                                            "This method should only be called after adapter established that the result is non-empty."));
+        }
+
+        @Override
+        public Collection<String> getCompatibleWriterStateNames() {
+            return sink.getCompatibleStateNames();
+        }
+    }
+
+    private class GlobalCommittingSinkAdapter extends TwoPhaseCommittingSinkAdapter
+            implements WithPostCommitTopology<InputT, CommT> {
+
+        @Override
+        public void addPostCommitTopology(DataStream<CommittableMessage<CommT>> committables) {
+            StandardSinkTopologies.addGlobalCommitter(committables, GlobalCommitterAdapter::new);
+        }
+    }
+
+    private class StatefulTwoPhaseCommittingSinkAdapter extends StatefulSinkAdapter

Review comment:
       have you considered using one Adapter class to implement all of these interfaces so that there are less decorators and therefore improve the code readability? The code in `SinkV1Adapter#asSpecializedSink()` can also be reduced appropriately.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittables;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.Committables;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link WriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends Committables<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (Committables<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittables<CommT> checkpoint :
+                committableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId)) {
+            // wait for all committables of the current checkpoint before submission

Review comment:
       the comment is a little bit confused. What does a boolean check mean "wait"?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java
##########
@@ -131,6 +150,10 @@ protected DataStreamSink(DataStream<T> inputStream, Sink<T, ?, ?, ?> sink) {
      */
     @PublicEvolving
     public DataStreamSink<T> setUidHash(String uidHash) {
+        if (!(transformation instanceof LegacySinkTransformation)) {

Review comment:
       It is the transformation's responsibility to handle this exception, i.e. in this case `SinkTransformation`

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -924,5 +924,10 @@ public long getDefaultBufferTimeout() {
         public ReadableConfig getGraphGeneratorConfig() {
             return config;
         }
+
+        @Override
+        public Collection<Integer> transform(Transformation<?> transformation) {

Review comment:
       Normally, Context, like the name describes, does not do actions




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e16a9d25a524114b579f05e2872949068588f720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r791771156



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.

Review comment:
       It seems that there are now two meanings of the word "comittables", after the new interface `Committables` has been created. Afaik, "commitables" has been used as a conceptual name previously very often. There is no corresponding domain entity developed for it and just let it open for users to define via the generic type `CommT`.
   
   Since all methods in this interface are working with subclasses of `ComittableMessage`, this interface is actually responsible for managing all commitables and all kinds of `ComittableMessage`, i.e. `ComittableSummary` and `ComittableWithLineage`, which turns out that calling this interface as `ComittablesManager` would be alternative naming option that is easier to understand the concept behind.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617",
       "triggerID" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ade991fe75b20a8c5403d6d4faa6b03aa105f207 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610) 
   * 50a8e4ce72502bcf3849c398cd8aed39da4b4f27 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796455508



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;

Review comment:
       Hi Fabian~ Thanks for the explanation! My main concern comes from that I think currently the `CheckpointCommittableManagerImpl` takes two roles: the first is the manager of committables of one checkpoint in memory and another is the data to be stored in the state. For the first role, having `subtaskId` and `numberOfSubtassks` is natural. But for the second role, the two fields are not needed, currently we set it with arbitrary values (0 and 1 respectively), I think this might cause a bit confusion. Thus it seems to me it might be better if we either extract out the two optional fields, or we do not reuse the same object for these two roles~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e16a9d25a524114b579f05e2872949068588f720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595) 
   * ade991fe75b20a8c5403d6d4faa6b03aa105f207 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027672049


   For the parallelism issue, is it possible that 
   1. Before call pre-committer, first modify the default parallelism of env to -1
   2. After the call finished, change the default parallelism back.
   
   Then we could know if the parallelism is set and we do not need to change the current behavior of the Transformations ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink 25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794360373



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/WriterOperator.java
##########
@@ -56,105 +59,64 @@
  * same parallelism or send them downstream to a {@link CommitterOperator} with a different
  * parallelism.
  *
- * <p>The operator may be part of a sink pipeline and is the first operator. There are currently two
- * ways this operator is used:
- *
- * <ul>
- *   <li>In streaming mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
+ * <p>The operator may be part of a sink pipeline and is the first operator.
  *
  * @param <InputT> the type of the committable
  * @param <CommT> the type of the committable (to send to downstream operators)
- * @param <WriterStateT> the type of the writer state for stateful sinks
  */
-class SinkOperator<InputT, CommT, WriterStateT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<InputT, byte[]>, BoundedOneInput {
+class WriterOperator<InputT, CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>

Review comment:
       From the code readability perspective, first of all, the package info is "hidden" in the import corner; second, it is not consistent, `Committer` -> `CommitterOperator`, but `SinkWriter` -> `WriterOperator`. Comparing to "Committer", the word "Writer" is more common and abstracted, could be used in many different scenarios. But, your point is also fair enough.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793588213



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {
+        this.subtasksCommittables = checkNotNull(subtasksCommittables);
+        // dummy collector, never used
+        this.collector = new CommittableCollector<>(0, 1);
+        this.checkpointId = checkpointId;
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return checkpointId;
+    }
+
+    Collection<SubtaskCommittables<CommT>> getSubtasks() {
+        return subtasksCommittables.values();
+    }
+
+    void addSummary(CommittableSummary<CommT> summary) {

Review comment:
       I think from a caller perspective it adds the summary and upsert would show some implementation detail.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794334300



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittables.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.Iterator;
+import java.util.List;
+import java.util.OptionalLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SubtaskCommittables<CommT> {

Review comment:
       > Formerly it is not required to have a doc string for `package-private` classes.
   
   I feel that SubtaskCommittable is one of the most important implementation classes. TBH, `package-privat` should not be the criteria for adding java doc or not. Java doc is required if the code can not explain itself or it will take too much time to read through the code, even it is a private method.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794287620



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -924,5 +924,10 @@ public long getDefaultBufferTimeout() {
         public ReadableConfig getGraphGeneratorConfig() {
             return config;
         }
+
+        @Override
+        public Collection<Integer> transform(Transformation<?> transformation) {

Review comment:
       That is definitely not the part I am most proud of :) Unfortunately, I need a way in a `SinkTransformationTranslator` to enqueue new transformations for the translation to support the dynamic expansion in case a sink uses a custom topology. 
   
   The only thing exposed is the `Context` in `translateForStreaming` and `translateForBatch`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1024099669


   @gaoyunhaii thanks for your review. 
   
   > I have one major concern regarding currently we might add the expanded transformations to the original environment. This seems to might cause problems if users call execute multiple times with the same env like
   > 
   > ```
   > DataStream a = ...
   > DataStream b = ...
   > b.addSink(new xx());
   > 
   > // The following would first clear all the current
   > // transformations, but when translating the sink,
   > // it would add the expanded transformations here.
   > env.execute()
   > 
   > b.map(...);
   > // Then here it would execute a graph also with 
   > // a -> some sink transformations
   > env.execute()
   > ```
   > 
   > Is it possible we use a separate `env` for the sink translation ? We could wrap the input transformation with a DataStream in the new environment, and at last we get all the transformations and transform them. We could also disable some methods inside adding topology, like switch runtime mode or configure slot resources.
   
   Maybe I overlooked something but I thought after every `StreamExecutionEnvironment#execute` the transformations are cleared. [1][2][3]
   
   Regarding your failover region comment https://github.com/apache/flink/pull/18428#discussion_r794246463 (for some reason I cannot answer your comment in the UI...) I assumed that the Batch exchange will never have an effect in streaming mode and therefore we could handle them equally. If that is not the case I also tend to do your suggestion.
   
   [1] https://github.com/apache/flink/blob/1704b1336bcbcdd42405f5e6e3e09c0176925210/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L2137
   [2] https://github.com/apache/flink/blob/1704b1336bcbcdd42405f5e6e3e09c0176925210/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L2120
   [3] https://github.com/apache/flink/blob/1704b1336bcbcdd42405f5e6e3e09c0176925210/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L1950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3b172e6de001227e6e383b5e7431eed5b529f8be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389) 
   * 3160e1d83687a15d9a602922aecc2d38f6891afa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] Airblader commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793785324



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       It's merged now. Sorry for that!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794354167



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterStateHandler.java
##########
@@ -17,34 +17,27 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.runtime.state.StateInitializationContext;
-import org.apache.flink.util.function.FunctionWithException;
-
-import java.io.Serializable;
-import java.util.List;
 
 /**
  * Manages the state of a {@link org.apache.flink.api.connector.sink.SinkWriter}. There are only two
  * flavors: stateless handled by {@link StatelessSinkWriterStateHandler} and stateful handled with
  * {@link StatefulSinkWriterStateHandler}.
  *
- * @param <WriterStateT>
+ * @param <InputT> the input type
  */
-interface SinkWriterStateHandler<WriterStateT> extends Serializable {
-    /**
-     * Extracts the writer state from the {@link StateInitializationContext}. The state will be used
-     * to create the writer.
-     */
-    List<WriterStateT> initializeState(StateInitializationContext context) throws Exception;
+interface SinkWriterStateHandler<InputT> {
 
     /**
-     * Stores the state of the supplier. The supplier should only be queried once.
+     * Stores the state of the writer.
      *
-     * @param stateExtractor
-     * @param checkpointId
+     * @param checkpointId the checkpointId
      */
-    void snapshotState(
-            FunctionWithException<Long, List<WriterStateT>, Exception> stateExtractor,
-            long checkpointId)
+    void snapshotState(long checkpointId) throws Exception;
+
+    /** Creates a writer, potentially using state from {@link StateInitializationContext}. */
+    SinkWriter<InputT> createWriter(InitContext initContext, StateInitializationContext context)

Review comment:
       The class name only tells that it is responsible for handling the state.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793812428



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface Committables<CommT> {
+    /** Returns a summary of the current commit progress. */
+    CommittableSummary<CommT> getSummary();
+
+    /**
+     * Commits all due committables.
+     *
+     * @param fullyReceived only commit committables if all committables of this checkpoint for a

Review comment:
       I am not sure whether I understood your idea correctly, for example: 
   
   @param fullyReceived gives the information whether all committables for the checkpoint are received, please refer to {@link CommitterOperator} for more information. Implementation class could use it with its own logic to determine the pending comitRequests that need to be committed, for example to check further if subtasksCommittableManager has received all commitables.   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1024508742


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796601239



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableManager.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface CommittableManager<CommT> {

Review comment:
       I agree the implementation handles both similarly but the interfaces separate both concerns. Since the operators are in a separate module I think it is a good idea to only expose the interactions through interfaces and do not expose the implementation. 
   
   Having two interfaces allows us in the future to maybe refactor the manager implementation and have different implementations for batch and streaming.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62e100ee426371e3cf041d52fcbf7d5b09af63d3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566) 
   * b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e16a9d25a524114b579f05e2872949068588f720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595) 
   * ade991fe75b20a8c5403d6d4faa6b03aa105f207 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610) 
   * 50a8e4ce72502bcf3849c398cd8aed39da4b4f27 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795720433



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link SinkWriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends CommittableManager<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (CommittableManager<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittableManager<CommT> manager :

Review comment:
       > Thanks @fapaul for the explanation! I think perhaps `CommittableSummary` is also the subclass of the the `CommittableMessage` and it should be able to be mixed in the returned list? And passing `committer` is indeed required, but it seems fine from my side~
   > 
   Good point. To be honest I like the current attraction that we have one central place to coordinate the additions and removals of the `CommittableMessage`: the `CommittableCollector` (I am open for a different name). In addition, we have two different interfaces to actually interact with the committables one is `CommittableManager` that is relevant to commit committables in batch execution mode, and `CheckpointCommittableManager` is used in streaming execution mode to commit committables.
   
   
   
   > Also may I have a double confirmation that It seems with the current implementation we'll emit `CommittableSummary` whenever we commit some committables instead of only when checkpointing? It seems to me it is to deal with the case that for unaligned checkpoint, the barrier might jump over all the summary and committable messages~ It should be also the reason we support receive multiple summary from one subtask for the same checkpoint, do I understand right~?
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027672049


   For the parallelism issue, is it possible that 
   1. Before call pre-committer topology, we first modify the default parallelism of env to -1
   2. After the call finished, change the default parallelism back.
   
   Then we could know if the parallelism is set and we do not need to change the current behavior of the Transformations ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027188780


   Hi Fabian @fapaul ~ Very thanks for the update! As a whole I'm also agree with that we could move some issues to separate PRs to unblock the following tasks, especially those issues only related to the internal implementation. 
   
   Currently some main remaining issues from my side are as follows:
   
   **1. About setParallelism:** 
   
   I think the current solution to the setParallelism issue is creative, but since it affects the result of `Transformation#getParallelism`, I'm a bit concern on if we have enumerate all the cases. Currently there are two possible issues from my side:
   
   - The semantics of `DataStream#getParallelism` is changed: previously it would always not return -1. This might be fixed by return the env's default if the transformation's parallelism is -1. 
   - `TableOperatorWrapperGenerator#calcParallelismAndResource` might have issues if it deals with both transformations with default parallelism and configured parallelism. Sorry I do not see an easy solution to solve this issue.
   
   I tried to see if we could first workaround this issue, but sorry I still not be able to find solutions.
   
   **2. About the repeat transformation**
   
   For the repeat transformation problem, I'm a bit hesitate that why we not remove the newly added transformations instead of the `SinkTransformation`? For example, it is feasible for users to first call `getExecutionPlan()` and then continue to change the properties of the `SinkTransformation`... 
   
   **3. About the commit protocol**
   
   Currently I'm a bit concern if we should keep there are only one summary message for each (checkpoint, subtask) pair, or we allow multiple messages but they must have the save total count. The problem here is that if we allow update the total count, the following operators would not have a method to know if it has received all the committables for one checkpoint. 
   
   Is it possible for now we first throws exception if the `CommitOperator` received the repeat summary message, but keeps the current implementation that it is still possible to emit multiple summary message for a single checkpoint? This poses stricter requirements for `pre-committer toplogy` and `post-committer topology`, thus it would not cause compatibility problem even we change the behavior afterwards.
   
   Also I have tried to think whether unaligned checkpoint + rescaling might cause problems, but currently I do not find issues.
   
   **4. About the PartitionTransformation (Optional)**
   
   > Your suggestion is definitely possible but it does not work for exchanges in the sink topology provided by the user. In this case, I would need to move the current logic from PartitionTransformationTranslator to SinkTransformationTranslator. Perhaps  it is a more general issue and the fix should reside in PartitionTransformationTranslator rather than only in the sink translation.
   >  Is there a downside when adding it to the PartitionTransformationTranslator?
   
   I think changing the partition type on-the-fly might cover the possible issues on the callers. I think `PartitionTransformationTranslator` could check and throw exception for impossible cases, but he should not modify it directly. The callers should ensure to avoid the impossible cases. This issue might be postponed since it only related to the implementation for now. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795511412



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link SinkWriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends CommittableManager<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (CommittableManager<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittableManager<CommT> manager :

Review comment:
       Thanks @fapaul for the explanation! I think perhaps `CommittableSummary` is also the subclass of the the `CommittableMessage`  and it should be able to be mixed in the returned list? And passing `committer` is indeed required, but it seems fine from my side~
   
   Also may I have a double confirmation that It seems with the current implementation we'll emit `CommittableSummary` whenever we commit some committables instead of only when checkpointing? It seems to me it is to deal with the case that for unaligned checkpoint, the barrier might jump over all the summary and committable messages~ It should be also the reason we support receive multiple summary from one subtask for the same checkpoint, do I understand right~?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 11c3547909dc1a430a00a9848c6ad9e36ff372b3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533) 
   * 62e100ee426371e3cf041d52fcbf7d5b09af63d3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793582242



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {
+        if (message instanceof CommittableSummary) {
+            addSummary((CommittableSummary<CommT>) message);
+        } else if (message instanceof CommittableWithLineage) {
+            addCommittable((CommittableWithLineage<CommT>) message);
+        }
+    }
+
+    /**
+     * Returns all {@link CheckpointCommittables} until the requested checkpoint id.
+     *
+     * @param checkpointId counter
+     * @return collection of {@link CheckpointCommittables}
+     */
+    public Collection<? extends CheckpointCommittables<CommT>> getCheckpointCommittablesUpTo(
+            long checkpointId) {
+        // clean up fully committed previous checkpoints
+        // this wouldn't work with concurrent unaligned checkpoints
+        Collection<CheckpointCommittablesImpl<CommT>> checkpoints =
+                checkpointCommittables.headMap(checkpointId, true).values();
+        checkpoints.removeIf(CheckpointCommittablesImpl::isFinished);
+        return checkpoints;
+    }
+
+    /**
+     * Returns all {@link CheckpointCommittables} that are currently hold by the collector.
+     *
+     * @return collection of {@link CheckpointCommittables}
+     */
+    @Nullable
+    public Collection<? extends CheckpointCommittables<CommT>> getEndOfInputCommittables() {
+        return getCheckpointCommittablesUpTo(EOI);
+    }
+
+    /**
+     * Returns whether all {@link CheckpointCommittables} currently hold by the collector are either
+     * committed or failed.
+     *
+     * @return state of the {@link CheckpointCommittables}
+     */
+    public boolean isFinished() {
+        return checkpointCommittables.values().stream()
+                .allMatch(CheckpointCommittablesImpl::isFinished);
+    }
+
+    /**
+     * Merges all information from an external collector into this collector.
+     *
+     * <p>This method is important during recovery from existing state.
+     *
+     * @param cc other {@link CommittableCollector}
+     */
+    public void merge(CommittableCollector<CommT> cc) {
+        for (Entry<Long, CheckpointCommittablesImpl<CommT>> checkpointEntry :
+                cc.checkpointCommittables.entrySet()) {
+            checkpointCommittables.merge(
+                    checkpointEntry.getKey(),
+                    checkpointEntry.getValue(),
+                    CheckpointCommittablesImpl::merge);
+        }
+    }
+
+    /**
+     * Returns number of subtasks.
+     *
+     * @return number of subtasks
+     */
+    public int getNumberOfSubtasks() {
+        return numberOfSubtasks;
+    }
+
+    /**
+     * Returns subtask id.
+     *
+     * @return subtask id.
+     */
+    public int getSubtaskId() {
+        return subtaskId;
+    }
+
+    /**
+     * Returns a new committable collector that deep copies all internals.
+     *
+     * @return {@link CommittableCollector}
+     */
+    public CommittableCollector<CommT> copy() {
+        return new CommittableCollector<>(
+                checkpointCommittables.entrySet().stream()
+                        .map(e -> Tuple2.of(e.getKey(), e.getValue().copy()))
+                        .collect(Collectors.toMap((t) -> t.f0, (t) -> t.f1)),
+                subtaskId,
+                numberOfSubtasks);
+    }
+
+    Collection<CheckpointCommittablesImpl<CommT>> getCheckpointCommittables() {
+        return checkpointCommittables.values();
+    }
+
+    private void addSummary(CommittableSummary<CommT> summary) {
+        checkpointCommittables
+                .computeIfAbsent(
+                        summary.getCheckpointId().orElse(EOI),
+                        key ->
+                                new CheckpointCommittablesImpl<>(
+                                        this, summary.getCheckpointId().orElse(EOI)))
+                .addSummary(summary);
+    }
+
+    private void addCommittable(CommittableWithLineage<CommT> committable) {
+        getCheckpointCommittables(committable).addCommittable(committable);
+    }
+
+    private CheckpointCommittablesImpl<CommT> getCheckpointCommittables(
+            CommittableMessage<CommT> committable) {

Review comment:
       If I change the signature of the method I need to expose the `EOI` (end of input) checkpoint number to the outside. I think it is best to keep it contained in this class.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898) 
   * 7e814ab74613f30066a09b7b7e7c406493607c58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [Flink-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f799eeea34b4d3c5691cc4b77f5be103ef1c065c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815) 
   * 337e1c36ebfc82b2e6056f81690d73382181eb88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793582809



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    private final NavigableMap<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables;
+    private final int subtaskId;
+    private final int numberOfSubtasks;
+
+    CommittableCollector(int subtaskId, int numberOfSubtasks) {
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+        this.checkpointCommittables = new TreeMap<>();
+    }
+
+    /** For deserialization. */
+    CommittableCollector(Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables) {
+        this(checkpointCommittables, 0, 1);
+    }
+
+    /** For deep-copy. */
+    CommittableCollector(
+            Map<Long, CheckpointCommittablesImpl<CommT>> checkpointCommittables,
+            int subtaskId,
+            int numberOfSubtasks) {
+        this.checkpointCommittables = new TreeMap<>(checkNotNull(checkpointCommittables));
+        this.subtaskId = subtaskId;
+        this.numberOfSubtasks = numberOfSubtasks;
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} based on the current runtime information. This method
+     * should be used for to instantiate a collector for all Sink V2.
+     *
+     * @param context holding runtime of information
+     * @param <CommT> type of the committable
+     * @return {@link CommittableCollector}
+     */
+    public static <CommT> CommittableCollector<CommT> of(RuntimeContext context) {
+        return new CommittableCollector<>(
+                context.getIndexOfThisSubtask(), context.getNumberOfParallelSubtasks());
+    }
+
+    /**
+     * Creates a {@link CommittableCollector} for a list of committables. This method is mainly used
+     * to create a collector from the state of Sink V1.
+     *
+     * @param r list of committables
+     * @param <CommT> type of committables
+     * @return {@link CommittableCollector}
+     */
+    static <CommT> CommittableCollector<CommT> ofLegacy(List<CommT> r) {
+        CommittableCollector<CommT> committableCollector = new CommittableCollector<>(0, 1);
+        // add a checkpoint with the lowest checkpoint id, this will be merged into the next
+        // checkpoint data, subtask id is arbitrary
+        CommittableSummary<CommT> summary =
+                new CommittableSummary<>(
+                        0, 1, InitContext.INITIAL_CHECKPOINT_ID, r.size(), r.size(), 0);
+        committableCollector.addSummary(summary);
+        SubtaskCommittables<CommT> subtask =
+                committableCollector
+                        .getCheckpointCommittables(summary)
+                        .getSubtaskCommittables(summary.getSubtaskId());
+        r.forEach(subtask::add);
+        return committableCollector;
+    }
+
+    /**
+     * Adds a {@link CommittableMessage} to the collector to hold it until emission.
+     *
+     * @param message either {@link CommittableSummary} or {@link CommittableWithLineage}
+     */
+    public void addMessage(CommittableMessage<CommT> message) {

Review comment:
       I'd prefer to keep it short.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796533774



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -7,7 +7,7 @@
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *     http://www.apache.org/licenses/LICENSE-2.0

Review comment:
       Unfortunately every time I save it in the IDE (Intellij) it is automatically updated 🙈 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62e100ee426371e3cf041d52fcbf7d5b09af63d3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027043787


   @gaoyunhaii Thanks for all your help 🙏  From my perspective the last blocking feature about the parallelism is fixed and in regards to the approaching release of Flink 1.15 do you think we can settle with a solution for the internals or maybe open a ticket to fix it after the feature freeze? 
   
   Currently, this PR blocks the migration of the existing sinks inside of Flink that would be great to get it done in 1.15 as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r796853042



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {

Review comment:
       Thanks for the explanation! Let's keep the current names for now~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793659498



##########
File path: flink-architecture-tests/violations/5b9eed8a-5fb6-4373-98ac-3be2a71941b8
##########
@@ -93,7 +93,6 @@ org.apache.flink.connector.file.src.util.Pool.recycler(): Returned leaf type org
 org.apache.flink.connector.file.src.util.Utils.forEachRemaining(org.apache.flink.connector.file.src.reader.BulkFormat$Reader, java.util.function.Consumer): Argument leaf type org.apache.flink.connector.file.src.reader.BulkFormat$Reader does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions$JDBCExactlyOnceOptionsBuilder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
 org.apache.flink.connector.jdbc.JdbcExecutionOptions.builder(): Returned leaf type org.apache.flink.connector.jdbc.JdbcExecutionOptions$Builder does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated
-org.apache.flink.connector.jdbc.JdbcSink.exactlyOnceSink(java.lang.String, org.apache.flink.connector.jdbc.JdbcStatementBuilder, org.apache.flink.connector.jdbc.JdbcExecutionOptions, org.apache.flink.connector.jdbc.JdbcExactlyOnceOptions, org.apache.flink.util.function.SerializableSupplier): Argument leaf type org.apache.flink.util.function.SerializableSupplier does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated

Review comment:
       Thanks for the hint, I'll remove my change as I think the other PR is merged before this one.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794279946



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java
##########
@@ -40,32 +45,46 @@
 
     private final PhysicalTransformation<T> transformation;
 
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, StreamSink<T> operator) {
-        this.transformation =
-                (PhysicalTransformation<T>)
-                        new LegacySinkTransformation<>(
-                                inputStream.getTransformation(),
-                                "Unnamed",
-                                operator,
-                                inputStream.getExecutionEnvironment().getParallelism());
-    }
-
-    @SuppressWarnings("unchecked")
-    protected DataStreamSink(DataStream<T> inputStream, Sink<T, ?, ?, ?> sink) {
-        transformation =
-                (PhysicalTransformation<T>)
-                        new SinkTransformation<>(
-                                inputStream.getTransformation(),
-                                sink,
-                                "Unnamed",
-                                inputStream.getExecutionEnvironment().getParallelism());
+    protected DataStreamSink(PhysicalTransformation<T> transformation) {
+        this.transformation = checkNotNull(transformation);
+    }
+
+    static <T> DataStreamSink<T> forSinkFunction(
+            DataStream<T> inputStream, SinkFunction<T> sinkFunction) {
+        StreamSink<T> sinkOperator = new StreamSink<>(sinkFunction);
+        PhysicalTransformation<T> transformation =
+                new LegacySinkTransformation<>(
+                        inputStream.getTransformation(),
+                        "Unnamed",
+                        sinkOperator,
+                        inputStream.getExecutionEnvironment().getParallelism());
+        inputStream.getExecutionEnvironment().addOperator(transformation);
+        return new DataStreamSink<>(transformation);
+    }
+
+    static <T> DataStreamSink<T> forSink(DataStream<T> inputStream, Sink<T> sink) {
+        StreamExecutionEnvironment executionEnvironment = inputStream.getExecutionEnvironment();
+        SinkTransformation<T, T> transformation =
+                new SinkTransformation<>(
+                        inputStream,
+                        sink,
+                        inputStream.getTransformation(),
+                        inputStream.getType(),
+                        "Sink",
+                        executionEnvironment.getParallelism());
         inputStream.getExecutionEnvironment().addOperator(transformation);
+        return new DataStreamSink<>(transformation);
+    }
+
+    @Internal
+    public static <T> DataStreamSink<T> forSinkV1(
+            DataStream<T> inputStream, org.apache.flink.api.connector.sink.Sink<T, ?, ?, ?> sink) {
+        return forSink(inputStream, SinkV1Adapter.wrap(sink));
     }
 
     /** Returns the transformation that contains the actual sink operator of this sink. */
     @Internal
-    public Transformation<T> getTransformation() {
+    public Transformation<?> getTransformation() {

Review comment:
       I reverted that change it feels unncessary.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794446296



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/Committables.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.Collection;
+
+/**
+ * Internal wrapper to handle the committing of committables.
+ *
+ * @param <CommT> type of the committable
+ */
+@Internal
+public interface Committables<CommT> {
+    /** Returns a summary of the current commit progress. */
+    CommittableSummary<CommT> getSummary();
+
+    /**
+     * Commits all due committables.
+     *
+     * @param fullyReceived only commit committables if all committables of this checkpoint for a

Review comment:
       `fullyReceived` is more like a signal that committables coming from a certain subtask for a checkpointId should be committed although not all committables from this subtask/checkpointId have been received.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795704693



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to book-keep the committing progress across checkpoints and subtasks.
+ * It handles the emission of committables and the {@link CommittableSummary}.
+ *
+ * @param <CommT> type of committable
+ */
+@Internal
+public class CommittableCollector<CommT> {
+    private static final long EOI = Long.MAX_VALUE;
+    /** Mapping of checkpoint id to {@link CheckpointCommittableManagerImpl}. */
+    private final NavigableMap<Long, CheckpointCommittableManagerImpl<CommT>>
+            checkpointCommittables;
+    /** Denotes the subtask id the collector is running. */
+    private final int subtaskId;

Review comment:
       I'd prefer the current solution because during the lifetime of a `CommittableCollector` the `subtaskId` and `numberOfSubtassks` should never change. If I add them as parameters to the commit method it introduces room for errors.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r795720433



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
##########
@@ -17,95 +17,173 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer;
+import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
 
+import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.OptionalLong;
 
 import static org.apache.flink.util.IOUtils.closeAll;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 
 /**
  * An operator that processes committables of a {@link org.apache.flink.api.connector.sink.Sink}.
  *
- * <p>The operator may be part of a sink pipeline but usually is the last operator. There are
- * currently two ways this operator is used:
+ * <p>The operator may be part of a sink pipeline, and it always follows {@link SinkWriterOperator},
+ * which initially outputs the committables.
  *
- * <ul>
- *   <li>In streaming mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is a {@link SinkOperator} with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and this operator containing the {@link
- *       org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
- *
- * @param <InputT> the type of the committable
- * @param <OutputT> the type of the committable to send to downstream operators
+ * @param <CommT> the type of the committable
  */
-class CommitterOperator<InputT, OutputT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<byte[], byte[]>, BoundedOneInput {
+class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, CommittableMessage<CommT>>,
+                BoundedOneInput {
+
+    private static final long RETRY_DELAY = 1000;
+    private final SimpleVersionedSerializer<CommT> committableSerializer;
+    private final Committer<CommT> committer;
+    private final boolean emitDownstream;
+    private CommittableCollector<CommT> committableCollector;
+    private long lastCompletedCheckpointId = -1;
 
-    private final SimpleVersionedSerializer<InputT> inputSerializer;
-    private final CommitterHandler<InputT, OutputT> committerHandler;
-    private final CommitRetrier commitRetrier;
+    /** The operator's state descriptor. */
+    private static final ListStateDescriptor<byte[]> STREAMING_COMMITTER_RAW_STATES_DESC =
+            new ListStateDescriptor<>(
+                    "streaming_committer_raw_states", BytePrimitiveArraySerializer.INSTANCE);
+
+    /** The operator's state. */
+    private ListState<CommittableCollector<CommT>> committableCollectorState;
 
     public CommitterOperator(
             ProcessingTimeService processingTimeService,
-            SimpleVersionedSerializer<InputT> inputSerializer,
-            CommitterHandler<InputT, OutputT> committerHandler) {
-        this.inputSerializer = checkNotNull(inputSerializer);
-        this.committerHandler = checkNotNull(committerHandler);
-        this.processingTimeService = processingTimeService;
-        this.commitRetrier = new CommitRetrier(processingTimeService, committerHandler);
+            SimpleVersionedSerializer<CommT> committableSerializer,
+            Committer<CommT> committer,
+            boolean emitDownstream) {
+        this.emitDownstream = emitDownstream;
+        this.processingTimeService = checkNotNull(processingTimeService);
+        this.committableSerializer = checkNotNull(committableSerializer);
+        this.committer = checkNotNull(committer);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<CommittableMessage<CommT>>> output) {
+        super.setup(containingTask, config, output);
+        committableCollector = CommittableCollector.of(getRuntimeContext());
     }
 
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        committerHandler.initializeState(context);
-        // try to re-commit recovered transactions as quickly as possible
-        commitRetrier.retryWithDelay();
+        committableCollectorState =
+                new SimpleVersionedListState<>(
+                        context.getOperatorStateStore()
+                                .getListState(STREAMING_COMMITTER_RAW_STATES_DESC),
+                        new CommittableCollectorSerializer<>(committableSerializer));
+
+        if (context.isRestored()) {
+            committableCollectorState.get().forEach(cc -> committableCollector.merge(cc));
+            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
+            // try to re-commit recovered transactions as quickly as possible
+            commitAndEmitCheckpoints();
+        }
     }
 
     @Override
     public void snapshotState(StateSnapshotContext context) throws Exception {
         super.snapshotState(context);
-        committerHandler.snapshotState(context);
+        // It is important to copy the collector to not mutate the state.
+        committableCollectorState.update(Collections.singletonList(committableCollector.copy()));
     }
 
     @Override
     public void endInput() throws Exception {
-        committerHandler.endOfInput();
-        commitRetrier.retryIndefinitely();
+        Collection<? extends CommittableManager<CommT>> endOfInputCommittables =
+                committableCollector.getEndOfInputCommittables();
+        // indicates batch
+        if (endOfInputCommittables != null) {
+            do {
+                for (CommittableManager<CommT> endOfInputCommittable : endOfInputCommittables) {
+                    commitAndEmit(endOfInputCommittable, false);
+                }
+            } while (!committableCollector.isFinished());
+        }
     }
 
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
-        committerHandler.notifyCheckpointCompleted(checkpointId);
-        commitRetrier.retryWithDelay();
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
+        commitAndEmitCheckpoints();
+    }
+
+    private void commitAndEmitCheckpoints() throws IOException, InterruptedException {
+        for (CheckpointCommittableManager<CommT> manager :

Review comment:
       > Thanks @fapaul for the explanation! I think perhaps `CommittableSummary` is also the subclass of the the `CommittableMessage` and it should be able to be mixed in the returned list? And passing `committer` is indeed required, but it seems fine from my side~
   > 
   Good point. To be honest I like the current abstraction that we have one central place to coordinate the additions and removals of the `CommittableMessage`: the `CommittableCollector` (I am open for a different name). In addition, we have two different interfaces to actually interact with the committables one is `CommittableManager` that is relevant to commit committables in batch execution mode, and `CheckpointCommittableManager` is used in streaming execution mode to commit committables.
   
   WDYT?
   
   > Also may I have a double confirmation that It seems with the current implementation we'll emit `CommittableSummary` whenever we commit some committables instead of only when checkpointing? It seems to me it is to deal with the case that for unaligned checkpoint, the barrier might jump over all the summary and committable messages~ It should be also the reason we support receive multiple summary from one subtask for the same checkpoint, do I understand right~?
   
   Yes, you are right considering unaligned checkpoints made it a bit more tricky. Do you think it needs more comments somewhere?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   * 327dbba3b2ee9dde5c70d73061fbc835eb051137 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1025891089


   > For the `execute` method, sorry I has some misunderstandings initially that the clear happens before compile happens, but it should indeed be happen after the generate() is called. However, there seems still cases that the transformation is not cleared, like users call `StreamExecutionEnvironment#getExecutionPlan()` ?
   
   Afaict `getExecutionPlan` is only used within tests and it feels to me like it is some part of Flink's legacy and is not used anymore. I will double-check that.
   
   Edit: I found a way to still support the use case. I remove the `SinkTransformation` after the expansion in the translation from the transformation list. This prevents double expansions.
   
   > For the Batch exchange, I think perhaps with branches (namely pass the runtime mode to SinkExpander and avoid create the failover region for streaming mode), we could avoid creating the batch exchange in advance for streaming mode, thus we won't need to rely on the behavior of the blocking edge under streaming mode~?
   
   Your suggestion is definitely possible but it does not work for exchanges in the sink topology provided by the user. In this case, I would need to move the current logic from `PartitionTransformationTranslator` to `SinkTransformationTranslator`. Perhaps it is a more general issue and the fix should reside in `PartitionTransformationTranslator` rather than only in the sink translation.
   Is there a downside when adding it to the `PartitionTransformationTranslator`?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794291984



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/WriterOperator.java
##########
@@ -56,105 +59,64 @@
  * same parallelism or send them downstream to a {@link CommitterOperator} with a different
  * parallelism.
  *
- * <p>The operator may be part of a sink pipeline and is the first operator. There are currently two
- * ways this operator is used:
- *
- * <ul>
- *   <li>In streaming mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and {@link
- *       org.apache.flink.api.connector.sink.Committer} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- *   <li>In batch mode, there is this operator with parallelism p containing {@link
- *       org.apache.flink.api.connector.sink.SinkWriter} and a {@link CommitterOperator} containing
- *       the {@link org.apache.flink.api.connector.sink.Committer} and {@link
- *       org.apache.flink.api.connector.sink.GlobalCommitter} with parallelism 1.
- * </ul>
+ * <p>The operator may be part of a sink pipeline and is the first operator.
  *
  * @param <InputT> the type of the committable
  * @param <CommT> the type of the committable (to send to downstream operators)
- * @param <WriterStateT> the type of the writer state for stateful sinks
  */
-class SinkOperator<InputT, CommT, WriterStateT> extends AbstractStreamOperator<byte[]>
-        implements OneInputStreamOperator<InputT, byte[]>, BoundedOneInput {
+class WriterOperator<InputT, CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>

Review comment:
       Usually, I am not a big fan to encode duplicate information in the java module path and class name. In this case, all the classes are already in the `sink` module and therefore belong to the `sink`. On the other, we also only call it `CommitterOperator` and not `SinkCommitterOperator`. 
   
   The information you are looking for should be part of the public-facing interfaces (`SinkWriter`, `Committer`) and I do not think it makes much sense to repeat them internally. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3b172e6de001227e6e383b5e7431eed5b529f8be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389) 
   * 3160e1d83687a15d9a602922aecc2d38f6891afa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419) 
   * e135c4f6208fb5efe5c0c0640f90d82f588e244d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingGe commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
JingGe commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r793778658



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittablesImpl.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.operators.sink.committables;
+
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class CheckpointCommittablesImpl<CommT> implements CheckpointCommittables<CommT> {
+    private final Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables;
+    private final CommittableCollector<CommT> collector;
+    private final long checkpointId;
+
+    CheckpointCommittablesImpl(CommittableCollector<CommT> collector, long checkpointId) {
+        this.collector = checkNotNull(collector);
+        this.checkpointId = checkpointId;
+        this.subtasksCommittables = new HashMap<>();
+    }
+
+    CheckpointCommittablesImpl(
+            Map<Integer, SubtaskCommittables<CommT>> subtasksCommittables, long checkpointId) {
+        this.subtasksCommittables = checkNotNull(subtasksCommittables);
+        // dummy collector, never used
+        this.collector = new CommittableCollector<>(0, 1);
+        this.checkpointId = checkpointId;
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return checkpointId;
+    }
+
+    Collection<SubtaskCommittables<CommT>> getSubtasks() {
+        return subtasksCommittables.values();
+    }
+
+    void addSummary(CommittableSummary<CommT> summary) {

Review comment:
       There is update logic in the add method, i.e. hidden magic code.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18428:
URL: https://github.com/apache/flink/pull/18428#discussion_r794373323



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
##########
@@ -7,7 +7,7 @@
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *     http://www.apache.org/licenses/LICENSE-2.0

Review comment:
       This was changed automatically by my IDE because I touched the file. In general, it seems our license includes this indentation [1], and probably the next one touching this file runs into the same issue. Do you think I should make a separate commit for this change? 
   
   [1] https://nightlies.apache.org/flink/flink-docs-master/docs/flinkdev/ide_setup/#copyright-profile




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586) 
   * e16a9d25a524114b579f05e2872949068588f720 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1027668111


   Hi Fabian~ 
   
   > I am not really familiar with this part of the code base but so far I did not see any failing tests regarding the TableOperatorWrapperGenerator can you elaborate a bit more why the current solution may cause problems.
   
   Sorry I'm also not expert in table / sql , but from the current implementation of TableOperatorWrapperGenerator, it tries to merge multiple transformations into one `MultipleInputTransformation`, and the parallelism of this one is set with the maximum of all the merged transformations. Previously the transformation might returns env.defaultParallelism in this step, but now it might return -1, suppose we have multiple parallelisms [200, env.default, env.default] and env.default = 300, previously the result would be 300 while now the result becomes 200.
   
   > Afaict the motivation to update the commit summary is exactly for unaligned checkpointing. In case the checkpoint barrier overtakes the records we only emit an incomplete summary and later need to emit a new summary with the updated amount of seen records.
   > In general, I am also open to restricting it in the first version and figuring out the unaligned checkpoint use cases later. WDYT?
   
   I think with unaligned checkpointing we should still be able to keep one summary for each checkpoint: each operator could only emit the summary after checkpoint complete notified & received all the summary from each input subtask? If this is feasible we could force the current implementation of pre-commit topology to follow this pattern, but not provide the same promise (only one summary to the same checkpont) to the post-commit topology, then we would not have compatibility no matter which pattern we want in the future~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18428: [FLINK-25575] Add Sink V2 operators and translation

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18428:
URL: https://github.com/apache/flink/pull/18428#issuecomment-1017677769


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29815",
       "triggerID" : "f799eeea34b4d3c5691cc4b77f5be103ef1c065c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29894",
       "triggerID" : "337e1c36ebfc82b2e6056f81690d73382181eb88",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29898",
       "triggerID" : "f4fdde6fd2c0cf28cf44d63bd8dfc34da24690d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29998",
       "triggerID" : "7e814ab74613f30066a09b7b7e7c406493607c58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9b29ad0b3697de70c570ee08d4ccf45f1c3e0bf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30008",
       "triggerID" : "1021113345",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30154",
       "triggerID" : "3705b617ac596bd4be08fd9ef2e4db40bef586f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30208",
       "triggerID" : "b3cd4cde01b0207349df086beb909b000b5e0bdd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a826e83de723921dd06636571743486cdef432b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30315",
       "triggerID" : "2a826e83de723921dd06636571743486cdef432b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30389",
       "triggerID" : "3b172e6de001227e6e383b5e7431eed5b529f8be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3160e1d83687a15d9a602922aecc2d38f6891afa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30419",
       "triggerID" : "1024508742",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30434",
       "triggerID" : "e135c4f6208fb5efe5c0c0640f90d82f588e244d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30525",
       "triggerID" : "327dbba3b2ee9dde5c70d73061fbc835eb051137",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30533",
       "triggerID" : "11c3547909dc1a430a00a9848c6ad9e36ff372b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30566",
       "triggerID" : "62e100ee426371e3cf041d52fcbf7d5b09af63d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30586",
       "triggerID" : "b9f6be0cce9e650cc6593ebcdd0fc300df32a2dd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e16a9d25a524114b579f05e2872949068588f720",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595",
       "triggerID" : "e16a9d25a524114b579f05e2872949068588f720",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610",
       "triggerID" : "ade991fe75b20a8c5403d6d4faa6b03aa105f207",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617",
       "triggerID" : "50a8e4ce72502bcf3849c398cd8aed39da4b4f27",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e16a9d25a524114b579f05e2872949068588f720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30595) 
   * ade991fe75b20a8c5403d6d4faa6b03aa105f207 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30610) 
   * 50a8e4ce72502bcf3849c398cd8aed39da4b4f27 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30617) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org