You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/25 22:32:32 UTC
[GitHub] [iceberg] stevenzwu commented on issue #2033: Iceberg specific files (manifest files, manifest list file, metadata file) are not created, only Flink specific manifest file is created
stevenzwu commented on issue #2033:
URL: https://github.com/apache/iceberg/issues/2033#issuecomment-767156860
I think one of the problem is how `IcebergFilesCommitter` use `maxCommittedCheckpointId` to de-dup committed manifests upon recovery. That is why it checkpoint and restore the checkpointId, which was impacted by the `Long.MAX_VALUE` with `endInput()`.
In our internal implementation, we are storing a hash value for each of the committing manifest file path. Upon recovery, we use that metadata in snapshot summary to filter out committed manifests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org