[GitHub] [iceberg] openinx commented on issue #2482: Error in writing CDC data with Flink

[GitBox <gi...@apache.org>]

openinx commented on issue #2482:
URL: https://github.com/apache/iceberg/issues/2482#issuecomment-826463184


   I checked the [place](https://github.com/apache/iceberg/blob/aba898b1a2ea15fd091228626b6887a5a72800c0/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L258) that throw the ValidationException.    The reason that we encountered this because we've expired the several latest snapshots, then there must be a snapshot whose parent snapshot is not null but we could not find the snapshot entry in `TableMetadata#snapshotById`.    
   
   For this issue, I think we should fix the problem from two aspects: 
   1.  The `validateDataFilesExist` should stop to traverse the null snapshot even if it's a parent of others. ( I will close the #2511 because it's not the correct direction I think )
   2.  The `ExpireSnapshotsSparkAction` will need to consider the format v2,  I'm still not sure whether will it be able to expire the snapshots in v2.    FYI @aokolnychyi 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org