You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Nikita Amelchev (Jira)" <ji...@apache.org> on 2024/11/29 18:25:00 UTC
[jira] [Updated] (IGNITE-21399) IgniteCache#getAll invoked on transactional cache can lead to a node failure with OutOfMemoryError
[ https://issues.apache.org/jira/browse/IGNITE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nikita Amelchev updated IGNITE-21399:
-------------------------------------
Release Note: Fixed logging of reduced cache entries for distributed long running operations.
> IgniteCache#getAll invoked on transactional cache can lead to a node failure with OutOfMemoryError
> --------------------------------------------------------------------------------------------------
>
> Key: IGNITE-21399
> URL: https://issues.apache.org/jira/browse/IGNITE-21399
> Project: Ignite
> Issue Type: Bug
> Reporter: Mikhail Petrov
> Assignee: Mikhail Petrov
> Priority: Major
> Labels: ise
> Fix For: 2.17
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Exception:
> {code:java}
> 2024-01-18 09:28:07.374 [ERROR][sys-#4371][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError]]
> java.lang.OutOfMemoryError: null
> at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:214) ~[?:?]
> at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:206) ~[?:?]
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:173) ~[?:?]
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538) ~[?:?]
> at java.lang.StringBuilder.append(StringBuilder.java:178) ~[?:?]
> at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?]
> at java.util.concurrent.ConcurrentHashMap.toString(ConcurrentHashMap.java:1329) ~[?:?]
> at java.lang.String.valueOf(String.java:2951) ~[?:?]
> at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils$18.toString(GridCacheUtils.java:646) ~[ignite-core-14.1.2.jar:14.1.2]
> at java.lang.String.valueOf(String.java:2951) ~[?:?]
> at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:102) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:910) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1121) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1055) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:622) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:561) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.future.GridCompoundFuture.toString(GridCompoundFuture.java:421) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.future.GridCompoundIdentityFuture.toString(GridCompoundIdentityFuture.java:47) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.processors.cache.distributed.dht.CacheDistributedGetFutureAdapter.toString(CacheDistributedGetFutureAdapter.java:377) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedGetFuture.toString(GridPartitionedGetFuture.java:664) ~[ignite-core-14.1.2.jar:14.1.2]
> at java.lang.String.valueOf(String.java:2951) ~[?:?]
> at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?]
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpLongRunningOperations0(GridCachePartitionExchangeManager.java:2290) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpLongRunningOperations(GridCachePartitionExchangeManager.java:2439) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.lambda$null$0(IgniteTxManager.java:398) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7437) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) ~[ignite-core-14.1.2.jar:14.1.2]
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) ~[ignite-core-14.1.2.jar:14.1.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:829) ~[?:?]
> 2024-01-18 09:28:07.411 [WARN ][sys-#4371][org.apache.ignite.internal.processors.cache.CacheDiagnosticManager] Page locks dump:
> {code}
> Java version: OpenJDK Runtime Environment 11.0.20+8-LTS
> IgniteCache#getAll is called on a transactional cache with a huge number of keys (> 1000000). Near node creates a future for this operation and sends requests for cache entries to remote nodes. All received entries are accumulated in the near node reducer (CacheDistributedGetFutureAdapter#rdc). If the mentioned future is not completed for a long time, long running operation detector will dump all pending futures in a log. The result includes mentioned reducer and string representation of all received cache entries.
> The `toString` implementation in the reducer just invokes ConcurrentHashMap#toString which can result in the OutOfMemoryError if huge amount of keys are requested. The OutOfMemoryError in this case does not mean that no free heap space is available (see https://github.com/openjdk/jdk/blob/d7ca08a5cc64c8d3941493d98423a49b5bc1b922/src/java.base/share/classes/java/lang/AbstractStringBuilder.java#L269)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)