You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/12/18 09:40:45 UTC
[GitHub] [iceberg] openinx opened a new issue #1959: Flink: Commit the existing manifests to iceberg txn in IcebergFilesCommitter.
openinx opened a new issue #1959:
URL: https://github.com/apache/iceberg/issues/1959
Currently, the flink's IcebergFilesCommitter will persist the recieved data files to flink manifests under a customized directory path ( in `snapshotState`), and then read all data files from the persist flink manifests to commit the iceberg txn ( in `notifyCheckpointComplete` ).
@stevenzwu suggested [1] to commit the written flink manifests to iceberg txn (by `appendManifests` or `appendDeleteManifests` API directly), which means we don't have to read the data files from persisted flink manifests and then construct the committing manifests again.
That sounds a great idea for the case that will commit lots files in a single TXN. I think it's good to create a separate issue to address that thing, let's track it here.
1. https://github.com/apache/iceberg/pull/1939#discussion_r545471359
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1959: Flink: Commit the existing manifests to iceberg txn in IcebergFilesCommitter.
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1959:
URL: https://github.com/apache/iceberg/issues/1959#issuecomment-748230293
I think it is a good idea to commit the manifests directly, but I would want a flag to tell Iceberg whether it should rewrite them. In some cases, manifests are correctly written into the metadata directory. But it should be possible to write a manifest in a different location. For example, we write manifests in a bucket in the local region and rewrite in the committer to create a manifest in the table's normal region.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org