You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "blcksrx (via GitHub)" <gi...@apache.org> on 2026/05/29 10:42:17 UTC
[PR] implement cache expiration policy [iceberg]
blcksrx opened a new pull request, #16610:
URL: https://github.com/apache/iceberg/pull/16610
### Allow dual catalog cache expiration policies (expire-after-access and expire-after-write)
---
This change enhances Iceberg’s catalog-level cache by allowing `expire-after-access` and `expire-after-write` policies to be used concurrently. This provides more flexible and powerful cache-tuning strategies by enabling both time-since-last-access and time-since-creation eviction policies on the same cache.
This is achieved by introducing a new property, `cache.expiration.expire-after-write-interval-ms`, which works in conjunction with the existing `cache.expiration-interval-ms` (which controls `expire-after-access`).
This addresses the pain point described in Issue #14417, where long-running streaming jobs could prevent cache entries from ever expiring under a pure access-based policy. By combining both policies, users can ensure periodic data refreshes while still efficiently caching frequently accessed tables.
---
#### Background
In many scenarios, especially with long-running services, you want to balance performance with data freshness. For example:
- Performance: Caching is essential to avoid the high cost of reloading table metadata for every query.
- Freshness: Cached entries must be periodically refreshed to pick up new snapshots or to prevent issues with expired credentials.
Using `expire-after-access` alone is not sufficient for continuous workloads, as frequently accessed entries may never be evicted. Using` expire-after-write` alone can be inefficient if it evicts "hot" entries that are still in active use.
By using both, you can configure a "best-of-both-worlds" strategy:
- A shorter expire-after-access duration to quickly evict inactive tables.
- A longer expire-after-write duration to act as a safety net, ensuring that even "hot" tables are refreshed periodically (e.g., before underlying credentials expire).
---
#### What changed in this PR
##### ✅ New catalog property for dual-policy expiration
Two distinct properties now control cache expiration. An entry is evicted if either condition is met. If both are set to a non-positive value, caching is disabled.
| Property | Default | Description |
| --------------------------------- | ------------------ | ------------------------------------------------------ |
| cache.expire-after-write-ms | 0 | Duration in milliseconds to expire a table from the cache after being created.tables will not refresh on write. 0 disables this policy. its disabled by default |
##### ✅ CachingCatalog supports dual policies
`CachingCatalog.wrap(...)` has been updated to configure the underlying Caffeine cache with both expiration policies when both properties are provided.
##### ✅ Tests cover dual-policy scenarios
Tests have been added to validate that when both policies are active, an entry is evicted based on whichever condition is met first, and that each policy works correctly on its own.
#####
Followup: #14440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] implement cache expiration policy [iceberg]
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx closed pull request #16610: implement cache expiration policy
URL: https://github.com/apache/iceberg/pull/16610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org