You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "madeirak (via GitHub)" <gi...@apache.org> on 2024/09/25 10:55:10 UTC
[I] What's the use of old metadata file， why not delete by default? [iceberg]
madeirak opened a new issue, #11206:
URL: https://github.com/apache/iceberg/issues/11206
### Query engine
Spark HiveCatalog
### Question
Every metadata file store the full amount of snapshot at that time, so why not delete metadata file by default?
User must manually `write.metadata.delete-after-commit.enabled` and `write.metadata.previous-versions-max`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] What's the use of old metadata file， why not delete by default? [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11206:
URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2462668198
Mostly for cases in which something has gone very very wrong. Snapshots don't take into account DML changes in schema or partition spec so it's impossible to fully revert state without having a metadata.json to go back to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] What's the use of old metadata file， why not delete by default? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11206:
URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2896113414
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] What's the use of old metadata file， why not delete by default? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11206: What's the use of old metadata file， why not delete by default?
URL: https://github.com/apache/iceberg/issues/11206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] What's the use of old metadata file， why not delete by default? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11206:
URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2856643163
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] What's the use of old metadata file， why not delete by default? [iceberg]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on issue #11206:
URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2374795119
Keeping old metadata helps support [rollback & time travel](https://iceberg.apache.org/docs/latest/spark-queries/#time-travel). It's often useful to know what the state of the table was at a certain point of time, or to be able to run a query against the table as it was at some point in time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] What's the use of old metadata file， why not delete by default? [iceberg]
Posted by "madeirak (via GitHub)" <gi...@apache.org>.
madeirak commented on issue #11206:
URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2375784911
> Keeping old metadata helps support [rollback & time travel](https://iceberg.apache.org/docs/latest/spark-queries/#time-travel). It's often useful to know what the state of the table was at a certain point of time, or to be able to run a query against the table as it was at some point in time.
Thx for reply, what if I have already deleted old data by expiring old snapshots, why is old metadata file ending in ".json" still retained?
Meanwhile, every new metadata file ending in ".json" contains full amount of snapshot information, rather than incremental information, why not delete old metadata file automatically？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org