[GitHub] [iceberg] pvary commented on issue #6630: Purpose of MAX_CONTINUOUS_EMPTY_COMMITS in IcebergFilesCommitter

["pvary (via GitHub)" <gi...@apache.org>]

pvary commented on issue #6630:
URL: https://github.com/apache/iceberg/issues/6630#issuecomment-1610804650

   The Flink snapshot/checpoint state is kept in 3 places:
   - Flink internal state - list of temp manifest files
   - File system - temp manifest files and uncomitted data files
   - Iceberg table data/metadata
   
   These 3 needs to be in sync, and we need to keep the changes since the last sync. So if we do not commit to Iceberg, then the Flink internal state and the file system temp tables are keep growing.  To avoid this, we commit from time to time (and write Flink metadata to the Iceberg table in the process), and after this commit we are able to remove old temp files and clean some data from the Flink state.
   
   So writing empty commit is intentional/needed, but the failure seems like a bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org