[GitHub] [iceberg] openinx commented on issue #2610: Flink CDC iceberg table have duplicate rows

[GitBox <gi...@apache.org>]

openinx commented on issue #2610:
URL: https://github.com/apache/iceberg/issues/2610#issuecomment-843743215


   What's your iceberg table schema?   Is it possible that the rows with the same primary key located in two different partitions/buckets ?   If sure,  then we may encounter the duplicated rows that from two different partitions/buckets because we won't dedup the same rows from different partitions , it is high cost to accomplish that. 
   
   Another thing,  all those change log events come from MYSQLbinlog or flink aggregate results ?  If they are came from binlog,  then each UPDATE will emit a `UPDATE_BEFORE` and a `UPDATE_AFTER`, currently we iceberg only support update (rather than UPSERT, that means we won't delete the existing rows  explicitily but we have pending PR to do this) rows,  in theory we won't encounter duplicate rows. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org