[GitHub] [iceberg] hbgstc123 commented on issue #3905: how to consume historical iceberg data with flink?

[GitBox <gi...@apache.org>]

hbgstc123 commented on issue #3905:
URL: https://github.com/apache/iceberg/issues/3905#issuecomment-1014196697


   @Initial-neko thanks for reply, if i use the current incremental consumption, I will have the following problem:
   1.I need to keep snapshots for a long time, which is costly and not necessary;
   2.There must not be delete or overwrite snapshots in between, as you mentioned; if delete or overwrite snapshots present, the result will be incorrect. Event if delete and overwrite snapshots are supported in the future, going through all the snapshots will still be inefficient since I only need the final result;
   
   Maybe a source that apply a partition filter on a latest snapshot of an iceberg table is a better solution for this scenario.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org