You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/02/17 00:57:08 UTC
[GitHub] [incubator-iceberg] rdblue commented on issue #780: expire table
metadata with ExpireSnapshots
rdblue commented on issue #780: expire table metadata with ExpireSnapshots
URL: https://github.com/apache/incubator-iceberg/pull/780#issuecomment-586777788
I think the purpose of this PR is to implement [a comment](https://github.com/apache/incubator-iceberg/pull/631#issuecomment-554168010) I made on #631:
> I think if we add expiration during ExpireSnapshots, then that's when the number of metadata versions would be pruned. For example, expiration might keep 10 previous metadata file references. Between expiration calls, the ones tracked continues to grow.
The open question is how this should behave. I'm not sure I agree with what I suggested the last time, with separate configurations for number to keep when cleaning up during expiration and for the maximum number to keep when writing metadata. If possible, it would be better to have just one configuration and let users choose whether to delete old metadata files as new ones are written (the current implementation) or to delete old metadata files during snapshot expiration.
I think we need to have the ability to delete old metadata files as new ones are created, or else tables with lots of transactions will suffer from larger and larger metadata files between snapshot expiration runs. I'm not so sure that we need to clean up old files during snapshot expiration. It would be nice to do it, but we don't want metadata file contents to grow too large because of this.
Right now, I'm leaning toward not doing this cleanup during snapshot expiration. It's additional complexity for not much benefit. What do you think, @yathindranath, @aokolnychyi, and @arina-ielchiieva?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org