[GitHub] [iceberg] openinx commented on a change in pull request #2863: Flink: Add streaming upsert write option.

[GitBox <gi...@apache.org>]

openinx commented on a change in pull request #2863:
URL: https://github.com/apache/iceberg/pull/2863#discussion_r704865520



##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -222,4 +222,7 @@ private TableProperties() {
 
   public static final String MERGE_CARDINALITY_CHECK_ENABLED = "write.merge.cardinality-check.enabled";
   public static final boolean MERGE_CARDINALITY_CHECK_ENABLED_DEFAULT = true;
+
+  public static final String UPSERT_MODE_ENABLE = "write.upsert.enable";
+  public static final boolean UPSERT_MODE_ENABLE_DEFAULT = false;

Review comment:
       > Is this only used in streaming mode now? Or does this work with Flink batch sink as well?
   
   Yes, it's only used for streaming mode right now. The batch upsert semantic has been implemented correctly by the [MERGE INTO](https://iceberg.apache.org/spark-writes/#merge-into) clause.
   
   > could we possibly use it to add our own support for CDC on top of Spark Structured Streaming?
   
   In theory, it's possible to add the CDC support for spark sturctured streaming, though the spark structured streaming does not support CDC event natively (I mean flink support INSERT/DELETE/UPDATE_BEFORE/UPDATE_AFTER events natively while Spark streaming doesn't unless we add extra field to indicate what's the operation type it is).  I think @XuQianJin-Stars @chenjunjiedada 's team are working on this issue in their own repo.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org