[GitHub] [iceberg] ben-manes commented on issue #3791: Deadlock when using CachingCatalog with Iceberg 0.13.0-SNAPSHOT

[GitBox <gi...@apache.org>]

ben-manes commented on issue #3791:
URL: https://github.com/apache/iceberg/issues/3791#issuecomment-1000053831


   Java's ConcurrentHashMap does not allow for computations to modify the hash map. This may livelock, deadlock, or may throw an exception (Java 9+).
   
   A `CacheWriter` performs its actions under a map computation to be atomic. This has since been deprecated and removed in 3.0. Your use-case of responding to evictions is either served by an `evictionListener` (atomic; notified within the `computeIfPresent` removing the entry) or `removalListener` (async; notified after the computation). Since your callback is removing more entries, it looks like an async removal listener might work better here.
   
   The `refreshAfterWrite` feature will proactively reload an entry if accessed after the threshold period. This can combined with expiration so that active content is kept fresh and inactive fades away. If only expiration is used then one may periodically see a latency spike when a popular item expires and the callers block waiting for it to be loaded. If both expiration and refresh are set to the same threshold then the entry is discarded, as one setting says it stale and the other that it is unusable. 
   
   The scheduler is helpful when application logic relies on timely removal of an expired entry. For example to close a websocket after the client times out due to inactivity. As the cache itself does not start any background threads, by default it relies on other activity (reads, writes) to trigger a maintenance cycle when it will sweep away expired entries.
   
   As an aside, I would recommend against using soft references as they have long been a problematic Java feature. The usual reason is a lack of knowledge of a good maximum size, but delegating to the GC can cause high pause times and churn. Instead improving instrumentation and setting a size threshold allows for more predictable latencies, avoids GC death spirals / OOME, and allows for use of a better eviction algorithm.
   
   hope that helps a little.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org