You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "victor (Jira)" <ji...@apache.org> on 2026/09/26 04:25:00 UTC

[jira] [Commented] (YARN-11994) EntityGroupFSTimelineStore: LRU cache eviction holds global lock causing all HTTP threads blocked and service hang

    [ https://issues.apache.org/jira/browse/YARN-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119577#comment-18119577 ] 

victor commented on YARN-11994:
-------------------------------

Attached {{YARN-11994.001.patch}} fixes the hang by moving cache eviction off the global lock. {{removeEldestEntry()}} no longer calls {{forceRelease()}} while holding the global {{SynchronizedMap}} monitor; it enqueues the evicted item, and a scheduled {{drainEvictionQueue()}} releases it on the executor thread outside the global lock.

Summary of the change:

* {{EntityGroupFSTimelineStore}}: async eviction queue + {{drainEvictionQueue()}} (1s fixed rate), drained on {{serviceStop}}.
* Test: {{TestEntityGroupFSTimelineStore#testAsyncEvictionReleasesStore}}.

Patch is based on apache/trunk {{90f0d1da37c}}. Would appreciate a review.


> EntityGroupFSTimelineStore: LRU cache eviction holds global lock causing all HTTP threads blocked and service hang
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-11994
>                 URL: https://issues.apache.org/jira/browse/YARN-11994
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: timelineservice
>    Affects Versions: 3.6.0
>            Reporter: victor
>            Priority: Major
>         Attachments: YARN-11994.001.patch
>
>
> h2. Problem
> {{EntityGroupFSTimelineStore}} (Timeline Service v1.5) keeps its LRU entity-group cache in a {{Collections.synchronizedMap(LinkedHashMap)}}. When the cache fills up, {{removeEldestEntry()}} evicts the eldest entry by calling {{EntityCacheItem.forceRelease()}} — a {{synchronized}} method — *while still holding the global {{SynchronizedMap}} monitor*:
> {code:java}
> protected boolean removeEldestEntry(...) {
>   if (super.size() > appCacheMaxSize) {
>     EntityCacheItem cacheItem = eldest.getValue();
>     cacheItem.forceRelease();  // synchronized — may block
>     return true;
>   }
>   return false;
> }
> {code}
> If another thread is inside {{refreshCache()}} (also {{synchronized}}, doing HDFS reads + JSON parsing + LevelDB writes that can take minutes), the eviction thread blocks on the {{EntityCacheItem}} lock while holding the global map lock — and every HTTP request thread that needs the cache then blocks on that global monitor. Observed in production on 3.4.1: 202 threads BLOCKED for 30+ minutes, ~20,000 CLOSE_WAIT connections, and file-descriptor exhaustion (32,768 limit) leading to complete service failure.
> h2. Root cause
> {{removeEldestEntry()}} runs inside {{SynchronizedMap.put()}}, so the eviction callback executes under the global map monitor, while {{refreshCache()}} is {{synchronized}} and holds the {{EntityCacheItem}} monitor across long I/O. A blocking {{forceRelease()}} therefore stalls on the {{EntityCacheItem}} lock while holding the global map lock, starving every other cache access.
> h2. Fix
> Evict asynchronously: {{removeEldestEntry()}} enqueues the evicted item onto a {{ConcurrentLinkedQueue}} and returns immediately, so {{put()}} no longer holds the global map lock while waiting on an {{EntityCacheItem}} lock. A scheduled {{drainEvictionQueue()}} (1s fixed rate) calls {{forceRelease()}} on the executor thread, outside the global map lock.
> h2. Related history
> YARN-4987 fixed this same read/evict race with {{refCount}}/{{tryRelease()}}, but YARN-5432 reverted it because {{tryRelease()}} re-created LevelDB stores at a fixed per-group path ("lock already held by another process"). YARN-9027's {{DB_COUNTER.getAndIncrement()}} unique path naming has since removed that conflict, so re-releasing an evicted group is safe.
> h2. Impact / Compatibility
> Eviction is now asynchronous with a ~1s drain window: an evicted item's store is closed by the drain thread shortly after eviction rather than synchronously. If the same group is evicted and immediately re-requested while its application is still running, the drain thread may reset the log offsets of an {{AppLogs}} still shared with the freshly re-created cache item, causing the next read to re-parse that group's logs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org