[GitHub] [flink] AHeise commented on a change in pull request #17870: [FLINK-24596][kafka] Fix buffered KafkaUpsert sink

[GitBox <gi...@apache.org>]

AHeise commented on a change in pull request #17870:
URL: https://github.com/apache/flink/pull/17870#discussion_r755789192



##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/ReducingUpsertSink.java
##########
@@ -43,19 +45,19 @@
     private final DataType physicalDataType;
     private final int[] keyProjection;
     private final SinkBufferFlushMode bufferFlushMode;
-    private final Function<RowData, RowData> valueCopyFunction;
+    @Nullable private final TypeSerializer<RowData> typeSerializer;
 
     ReducingUpsertSink(
             Sink<RowData, ?, WriterState, ?> wrappedSink,
             DataType physicalDataType,
             int[] keyProjection,
             SinkBufferFlushMode bufferFlushMode,
-            Function<RowData, RowData> valueCopyFunction) {

Review comment:
       Here we could also use `SerializableFunction` but your change is also good (in general not a huge fan of nullable fields that imply different behavior).

##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/SinkBufferFlushMode.java
##########
@@ -18,10 +18,11 @@
 
 package org.apache.flink.streaming.connectors.kafka.table;
 
+import java.io.Serializable;
 import java.util.Objects;
 
 /** Sink buffer flush configuration. */
-public class SinkBufferFlushMode {
+public class SinkBufferFlushMode implements Serializable {

Review comment:
       🙈 

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/collect/CollectStreamSink.java
##########
@@ -30,17 +32,19 @@
 @Internal
 public class CollectStreamSink<T> extends DataStreamSink<T> {
 
-    private final LegacySinkTransformation<T> transformation;
+    private final PhysicalTransformation<T> transformation;
 
+    @SuppressWarnings("unchecked")
     public CollectStreamSink(DataStream<T> inputStream, CollectSinkOperatorFactory<T> factory) {
         super(inputStream, (CollectSinkOperator<T>) factory.getOperator());
         this.transformation =
-                new LegacySinkTransformation<>(
-                        inputStream.getTransformation(), "Collect Stream Sink", factory, 1);
+                (PhysicalTransformation<T>)

Review comment:
       Why is the cast necessary?
   Can you try to make `public class LegacySinkTransformation<T> extends PhysicalTransformation<T>`?
   The second ctor should a factory method instead...

##########
File path: flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecSinkITCase.java
##########
@@ -152,6 +154,43 @@ public void invoke(
         assertTimestampResults(timestamps, rows);
     }
 
+    @Test
+    public void testUnifiedSinksAreUsableWithDataStreamSinkProvider()
+            throws ExecutionException, InterruptedException {
+        final StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
+        final SharedReference<List<RowData>> fetched = sharedObjects.add(new ArrayList<>());
+        final List<Row> rows = Arrays.asList(Row.of(1), Row.of(2));
+
+        final TableDescriptor sourceDescriptor =
+                TableFactoryHarness.newBuilder()
+                        .schema(Schema.newBuilder().column("a", INT()).build())
+                        .source(new TimestampTestSource(rows))
+                        .sink(
+                                new TableFactoryHarness.SinkBase() {
+                                    @Override
+                                    public DataStreamSinkProvider getSinkRuntimeProvider(
+                                            DynamicTableSink.Context context) {
+                                        return dataStream ->
+                                                dataStream.sinkTo(
+                                                        TestSink.newBuilder()
+                                                                .setWriter(
+                                                                        new RecordWriter(fetched))
+                                                                .setCommittableSerializer(
+                                                                        TestSink
+                                                                                .StringCommittableSerializer
+                                                                                .INSTANCE)
+                                                                .build());
+                                    }
+                                })

Review comment:
       Could you please extract that to a method local class to ease the formatting?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org