You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/08 06:14:27 UTC
[GitHub] [iceberg] ajantha-bhat opened a new issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots actions and call procedure ?
ajantha-bhat opened a new issue #3496:
URL: https://github.com/apache/iceberg/issues/3496
Currently `table.expireSnapshots()` will clean the data files and manifest files using `RemoveSnapshots#cleanExpiredSnapshots`.
But the Spark action `ExpireSnapshots`, always disable `cleanExpiredSnapshots`. we should provide a method to support it and also in the CALL procedure ?
If yes, please assign this issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots spark action and call procedure ?
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3496:
URL: https://github.com/apache/iceberg/issues/3496#issuecomment-963156998
The default behavior of the Action is to remove the files which are no longer live. The difference between the table api and the action, is that the Action determines the set of files to be removed by using a distributed job while the table api does this calculation locally.
The main reason behind writing the action was that the default api does not scale well for extremely large tables. To preserve the functionality of the original api we added the flag which causes the original api to just remove snapshots and not delete files. The Action then takes the difference in state between before and after running the api, and uses that information to delete the files.
This is explained in the Java Doc for the Class
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java#L54-L67
Here you can see the delete function is applied on the diff set result
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java#L214-L221
The default delete action removes the files see
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java#L82-L89
---
So I do not believe any changes are needed here. If a user did want to use a separate delete facility I would actually suggest they use
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java#L154
which was added explicitly for users who have some sort of distributed async delete solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat closed issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots spark action and call procedure ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat closed issue #3496:
URL: https://github.com/apache/iceberg/issues/3496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots spark action and call procedure ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3496:
URL: https://github.com/apache/iceberg/issues/3496#issuecomment-962987296
I know that we support `deleteWith` method in spark action to collect expired data file, manifest file and manifest list.
But user cannot delete it as it might be referenced by other live snapshots.
So as `RemoveSnapshots#cleanExpiredSnapshots` is already having the logic to find out files that are not referenced by any live snapshots. I think we should support enabling it from actions also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots spark action and call procedure ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3496:
URL: https://github.com/apache/iceberg/issues/3496#issuecomment-963163923
> Here you can see the delete function is applied on the diff set result
@RussellSpitzer : Yeah, that's my concern. we cannot just delete the files formed by diff set right? **Because those files can still be referred by new snapshots which are not expired.** I saw the logic to handle this problem in `RemoveSnapshots#cleanExpiredSnapshots`
But I didn't see that logic in Actions flow. So, how it is handled in actions flow ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots spark action and call procedure ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3496:
URL: https://github.com/apache/iceberg/issues/3496#issuecomment-963168188
got it. Thanks for the clarifications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots spark action and call procedure ?
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3496:
URL: https://github.com/apache/iceberg/issues/3496#issuecomment-963166556
It is handled by using the "ALL" versions of the respective metadata tables which check not just the set of live files, but also all files which are currently reachable by any known snapshot.
See
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java#L119
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java#L119
https://github.com/apache/iceberg/blob/9b285d049c094ca6ee717e159249dee36d118894/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java#L139-L140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3496: Enable RemoveSnapshots#cleanExpiredSnapshots from ExpireSnapshots actions and call procedure ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3496:
URL: https://github.com/apache/iceberg/issues/3496#issuecomment-962843440
cc: @aokolnychyi , @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org