[GitHub] [incubator-iceberg] bryanck commented on issue #200: Add SnapshotProducer that can rewrite manifests

[GitBox <gi...@apache.org>]

bryanck commented on issue #200: Add SnapshotProducer that can rewrite manifests
URL: https://github.com/apache/incubator-iceberg/pull/200#issuecomment-498414249
 
 
   This will create entirely new manifests (with a new snapshot ID and with new contents) that replace existing manifests, so in essence, yes, the manifests and contents will be reordered. We were planning to apply this API on a regular basis on new manifests (e.g. hourly or daily). Also, `commit.manifest.target-size-bytes` will not be respected thus this could generate large manifest files.
   
   I first thought of adding this functionality to `MergingSnapshotProducer` but this would either generate far too many manifests (trading long file scans for long manifest scans) or generate manifests with wide partition ranges as it does now.
   
   Respecting `commit.manifest.target-size-bytes` should be straightforward if that is desired. I'm not sure how respecting manifest order is possible though.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org