You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2025/06/12 01:30:57 UTC
[I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
slfan1989 opened a new issue, #13300:
URL: https://github.com/apache/iceberg/issues/13300
### Query engine
Spark
### Question
I noticed that in `SnapshotTableSparkAction`, the `gc.enabled` property is explicitly set to false. I'm trying to understand the reasoning behind this design choice. Could someone please explain why `gc.enabled` is disabled in this context? Is it to avoid unintended data removal during the snapshot operation, or are there other considerations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #13300:
URL: https://github.com/apache/iceberg/issues/13300#issuecomment-2964717624
@szehon-ho @manuzhang Do you know why this happens? Looking forward to your thoughts!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #13300:
URL: https://github.com/apache/iceberg/issues/13300#issuecomment-2967734133
As @jkolash noted, GC.Enabled is set to false because the "snapshot" essentially does not own the data files it references. If it deletes them it may corrupt the table it was snapshotted from. So we block that unless a user specifically knows what they are doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 closed issue #13300: Understand why SnapshotTableSparkAction disables gc.enabled by default
URL: https://github.com/apache/iceberg/issues/13300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
Posted by "jkolash (via GitHub)" <gi...@apache.org>.
jkolash commented on issue #13300:
URL: https://github.com/apache/iceberg/issues/13300#issuecomment-2966099141
The maintenence jobs that run normally may attempt to delete files at the snapshotted location. Which may be undesirable. I do actually snapshot the same tables repeatedly and turn this off. My recommendation if you want to do this is to disable file file deletion logic in maintenence jobs and rely only on orphan file deletion for deleting files. This will in effect only search orphan files inside the standard table location paths. This also speeds up maintenece jobs and the deletion cost is amortized with the orphan cleanup which will do the deletes in bulk. The tradeoff is you may be keeping files around longer than necessary between orphan cleanups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #13300:
URL: https://github.com/apache/iceberg/issues/13300#issuecomment-2968650617
@jkolash @RussellSpitzer Thank you for the explanation! I’ve understood your approach, and the feedback makes a lot of sense — it’s been very helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Understand why SnapshotTableSparkAction disables gc.enabled by default [iceberg]
Posted by "jkolash (via GitHub)" <gi...@apache.org>.
jkolash commented on issue #13300:
URL: https://github.com/apache/iceberg/issues/13300#issuecomment-2966103416
https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/ExpireSnapshots.java#L40 for example lets you supply a custom delete function which can be a no-op.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org