You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ricardopereira33 (via GitHub)" <gi...@apache.org> on 2025/03/25 16:35:29 UTC
[I] Rest Catalog: Remove snapshots more efficiently [iceberg]
ricardopereira33 opened a new issue, #12642:
URL: https://github.com/apache/iceberg/issues/12642
### Proposed Change
## Problem
We found an issue when expiring old snapshots from a table with a lot of snapshots (+10k). The issue happens when the `expireSnapshots` action triggers a request with a list of `UpdateTableRequest` that will clean up most of the snapshots (~99%). The Rest Catalog server will receive a single request (UpdateTableRequest), with the list of snapshots to be removed from the metadata. However, the metadata updates on each snapshot:
https://github.com/apache/iceberg/blob/03ff41c189c7420992be0e4a4ddc63f005e2e0d5/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L435
The `applyTo` method will basically just update the Table for 1 snapshot, even though it could receive the entire list of snapshots:
https://github.com/apache/iceberg/blob/03ff41c189c7420992be0e4a4ddc63f005e2e0d5/core/src/main/java/org/apache/iceberg/MetadataUpdate.java#L344
The `removeSnapshots` method delegates the process to the `rewriteSnapshotInternal`:
https://github.com/apache/iceberg/blob/03ff41c189c7420992be0e4a4ddc63f005e2e0d5/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1424
This `rewriteSnapshotInternal` iterates over all snapshots of the table to remove the provided list of snapshots (as I mentioned, **it only passes 1 snapshot**). This is not efficient, when we need to remove a huge amount of snapshots, we need to iterate over the entire list of snapshots (N elements), N times - O(Nˆ2).
We notice this issue when we recently enable some tables that are written by streaming jobs (they are often written and generate a lot of snapshots). With +10 tables, having +10k snapshots each, some of them +100k, cause our Rest Catalog server to hit 100% CPU usage constantly:

## Proposal
On the `CatalogHandlers`, we could group by the `MetadataUpdate`, and apply them in bulk:
https://github.com/apache/iceberg/blob/03ff41c189c7420992be0e4a4ddc63f005e2e0d5/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L435
### Proposal document
_No response_
### Specifications
- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Rest Catalog: Remove snapshots more efficiently [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12642:
URL: https://github.com/apache/iceberg/issues/12642#issuecomment-3507222336
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Rest Catalog: Remove snapshots more efficiently [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #12642:
URL: https://github.com/apache/iceberg/issues/12642#issuecomment-2873732823
@ricardopereira33 Thanks for working on this! Since the PR went in, should we close this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Rest Catalog: Remove snapshots more efficiently [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on issue #12642:
URL: https://github.com/apache/iceberg/issues/12642#issuecomment-2752362503
+1 to do it efficiently, this will help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Rest Catalog: Remove snapshots more efficiently [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12642: Rest Catalog: Remove snapshots more efficiently
URL: https://github.com/apache/iceberg/issues/12642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Rest Catalog: Remove snapshots more efficiently [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #12642:
URL: https://github.com/apache/iceberg/issues/12642#issuecomment-2751890851
I remember seeing this in the implementation and thinking the same thing, but it never surfaced as an issue for us. Probably because with continuous maintenance N was small so even if it was N^2 it didn't really end up making too much of a difference.
I think it makes total sense though to fix this (e.g. it's totally possible that snapshots accumulated and for whatever reason expiration couldn't run for a while, and I agree with the proposed fix we should just pass in the whole set of snapshots to remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Rest Catalog: Remove snapshots more efficiently [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12642:
URL: https://github.com/apache/iceberg/issues/12642#issuecomment-3567192987
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org