You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2019/08/04 20:04:08 UTC
[GitHub] [incubator-iceberg] aokolnychyi edited a comment on issue #309: Add
mechanism to expire table metadata
aokolnychyi edited a comment on issue #309: Add mechanism to expire table metadata
URL: https://github.com/apache/incubator-iceberg/pull/309#issuecomment-518031926
It is a great effort, we were starting to think about this as well.
One of the reasons why we started looking into Iceberg in the first place was the ability to avoid lists during job planning/commit. I would still avoid lists even for operational things if it doesn’t complicate the logic too much or impact the performance of other operations. If there is no better way, I think we can still use lists in this particular case. I would expect to see old files consistently but it might expensive.
A couple of questions:
- If we do use lists, how do we plan to handle cases when `write.metadata.path` changes over time? Am I correct that there is no guarantee that the current metadata location contains all files?
- Iceberg tables can share locations. Can they share metadata locations? If yes, how do we plan to handle it? (include table UUID in `metadata.json` files?)
- If we are to keep only 100 last files, does it mean that we will be deleting the most recent file on every commit?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org