[GitHub] [iceberg] rdblue commented on pull request #1515: Flink: Support sink when disable flink checkpoint disable

[GitBox <gi...@apache.org>]

rdblue commented on pull request #1515:
URL: https://github.com/apache/iceberg/pull/1515#issuecomment-707375751


   I think it would make sense to get a configuration check into 0.10.0 to ensure that users don't configure the sink without checkpointing. That could cause users to lose data.
   
   Before we commit this feature, though, I think we need to think through it a bit more. Is at-most-once a normal guarantee for Flink? Other systems I've seen typically guarantee at-least-once.
   
   Also, is there a way to recover the state of the incoming stream from Iceberg commit metadata? For example, what if data files are sent to the committer with the partition offsets that have been processed by the writer for the given task? Then a streaming job could recover by finding the last streaming commit and reconstructing the offsets for all partitions from metadata.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org