You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Sergey Korotkov (Jira)" <ji...@apache.org> on 2025/06/21 09:58:00 UTC
[jira] [Updated] (IGNITE-24992) Hang in put() and starvation in sys-striped pool if RANDOM_LRU or RANDOM_2_LRU eviction policy is used
[ https://issues.apache.org/jira/browse/IGNITE-24992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Korotkov updated IGNITE-24992:
-------------------------------------
Summary: Hang in put() and starvation in sys-striped pool if RANDOM_LRU or RANDOM_2_LRU eviction policy is used (was: Hang in put() and starvation in sys-striped pool if RANDOM_2_LRU eviction policy is used)
> Hang in put() and starvation in sys-striped pool if RANDOM_LRU or RANDOM_2_LRU eviction policy is used
> ------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-24992
> URL: https://issues.apache.org/jira/browse/IGNITE-24992
> Project: Ignite
> Issue Type: Bug
> Reporter: Sergey Korotkov
> Assignee: Sergey Korotkov
> Priority: Major
> Labels: ise
> Attachments: Random2LruPageEvictionPutLargeObjectsTest.java
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In-memory cluster.
> RANDOM_2_LRU or RANDOM_LRU eviction policy is applied.
> Put of large objects which occupy several pages can hang in cycle in the {{IgniteCacheDatabaseSharedManager.ensureFreeSpace()}} since {{Random2LruPageEvictionTracker.evictDataPage()}} keeps failing to find the page to evict.
> The immediate reason is that RANDOM_2_LRU approach can only evict pages "with at least one touch". For large (fragmented) objects only the last page is touched (see the {{PageEvictionTracker.touchPage()}} call in {{AbstractFreeList#WriteRowHandler.addRow()}} method). So if only large objects exist data region has very very small fraction of the "touched" pages appropriate for eviction. It appears that 5000 random attempts are not enough to get 5 candidate pages to evict. So Random2LruPageEvictionTracker.evictDataPage() fails.
> System striped pool can starvate for a long time - upto 14 hours once in real production environment until nodes were manually restarted.
> Reproducer is attached [^Random2LruPageEvictionPutLargeObjectsTest.java].
> It hangs after 12th put:
> {noformat}
> ...
> >>> Key put: 21, total entries put: 12
> [2025-04-02T16:34:23,108][WARN ][sys-stripe-7-#62%paged.Random2LruPageEvictionPutLargeObjectsTest1%][IgniteCacheDatabaseSharedManager] Page-based evictions started. Consider increasing 'maxSize' on Data Region configuration: default
> [2025-04-02T16:34:23,110][WARN ][sys-stripe-7-#62%paged.Random2LruPageEvictionPutLargeObjectsTest1%][Random2LruPageEvictionTracker] Too many attempts to choose data page: 5000
> ....
> [2025-04-02T16:34:48,277][ERROR][tcp-disco-msg-worker-[15e5f20b 127.0.0.1:47500]-#8%paged.Random2LruPageEvictionPutLargeObjectsTest1%-#92%paged.Random2LruPageEvictionPutLargeObjectsTest1%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=sys-stripe-7, threadName=sys-stripe-7-#62%paged.Random2LruPageEvictionPutLargeObjectsTest1%, blockedFor=25s]
> [2025-04-02T16:34:48,278][WARN ][tcp-disco-msg-worker-[15e5f20b 127.0.0.1:47500]-#8%paged.Random2LruPageEvictionPutLargeObjectsTest1%-#92%paged.Random2LruPageEvictionPutLargeObjectsTest1%][IgniteTestResources] Possible failure suppressed accordingly to a configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-7, igniteInstanceName=paged.Random2LruPageEvictionPutLargeObjectsTest1, finished=false, heartbeatTs=1743586463106]]]
> org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-7, igniteInstanceName=paged.Random2LruPageEvictionPutLargeObjectsTest1, finished=false, heartbeatTs=1743586463106]
> at org.apache.ignite.internal.processors.cache.persistence.evict.Random2LruPageEvictionTracker.evictDataPage(Random2LruPageEvictionTracker.java:152) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.ensureFreeSpace(IgniteCacheDatabaseSharedManager.java:1251) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1812) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1732) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3225) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$3.apply(GridDhtAtomicCache.java:271) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$3.apply(GridDhtAtomicCache.java:266) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1096) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:597) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:398) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:316) ~[classes/:?]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:306) ~[classes/:?]
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1877) ~[classes/:?]
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1498) ~[classes/:?]
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1402) ~[classes/:?]
> at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55) ~[classes/:?]
> at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:637) ~[classes/:?]
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) ~[classes/:?]
> at java.base/java.lang.Thread.run(Thread.java:829) ~[?:?]
> [2025-04-02T16:34:48,279][WARN ][tcp-disco-msg-worker-[15e5f20b 127.0.0.1:47500]-#8%paged.Random2LruPageEvictionPutLargeObjectsTest1%-#92%paged.Random2LruPageEvictionPutLargeObjectsTest1%][CacheDiagnosticManager] Page locks dump:
> [2025-04-02T16:34:48,661][WARN ][grid-timeout-worker-#76%paged.Random2LruPageEvictionPutLargeObjectsTest1%][PoolProcessor] >>> Possible starvation in striped pool.
> Thread name: sys-stripe-7-#62%paged.Random2LruPageEvictionPutLargeObjectsTest1%
> Queue: []
> Deadlock: false
> Completed: 2
> Thread [name="sys-stripe-7-#62%paged.Random2LruPageEvictionPutLargeObjectsTest1%", id=80, state=RUNNABLE, blockCnt=0, waitCnt=3]
> at app//o.a.i.i.processors.cache.persistence.evict.Random2LruPageEvictionTracker.evictDataPage(Random2LruPageEvictionTracker.java:152)
> at app//o.a.i.i.processors.cache.persistence.IgniteCacheDatabaseSharedManager.ensureFreeSpace(IgniteCacheDatabaseSharedManager.java:1251)
> at app//o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1812)
> at app//o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1732)
> at app//o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3225)
> at app//o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$3.apply(GridDhtAtomicCache.java:271)
> at app//o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$3.apply(GridDhtAtomicCache.java:266)
> at app//o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1096)
> at app//o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:597)
> at app//o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:398)
> at app//o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:316)
> at app//o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:306)
> at app//o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1877)
> at app//o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1498)
> at app//o.a.i.i.managers.communication.GridIoManager$9.execute(GridIoManager.java:1402)
> at app//o.a.i.i.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
> at app//o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:637)
> at app//o.a.i.i.util.worker.GridWorker.run(GridWorker.java:125)
> at java.base@11.0.26/java.lang.Thread.run(Thread.java:829)
> {noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)