[GitHub] [iceberg] hameizi commented on pull request #3095: Flink: flink read iceberg upsert data use streaming mode

[GitBox <gi...@apache.org>]

hameizi commented on pull request #3095:
URL: https://github.com/apache/iceberg/pull/3095#issuecomment-1017216735


   > 1. flink writing cdc data to iceberg will ignore UPDATE_BEFORE and treat UPDATE_AFTER as retract message,  it's completely not correct. The relevant codes are as follows:
   
   For this issue, config `write.upsert.enabled` is not suitable for CDC. So if your scene is CDC you can config it false that result `upsert` false.
   
   
   > 2\. when there is only delete operation during the period of one snapshot，the snapshot of iceberg will contains only equality delete files  and no data file(data file is deleted),  flink will ignore equality delete files and miss all  -D .   function org.apache.iceberg.ManifestGroup.planStreamingFiles()   create FileScanTask  only when data file exists, so flink can not process equality delete files.
   
   I will do more test for this issue.
   Thanks for your feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org