You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by jackluo923 <ja...@mail.utoronto.ca> on 2019/07/03 21:55:23 UTC
Ignite Durable Memory Runs out of Heap Space
Hi,
I am trying to store many kv pairs into ignite than the available ram.
At this moment, the primary purpose of using ignite is a distributed kv
database.
*My current configuration is: *
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="initialSize" value="#{500L * 1024 * 1024}"/>
<property name="maxSize" value="#{32L * 1024 * 1024 * 1024}"/>
</bean>
</property>
The physical machine has 64GB of ram. The data which I am trying to insert
into ignite is spread across multiple caches (around 20). From my current
understanding, we shouldn't receive any "Out of Heap Space" since the ignite
uses "off heap storage" and anything which does not fit in the allocated
cache space will be swapped to disk. Do I have any obvious flaw in my
understanding or incorrect configuration?
A small snippet of log message:
[01:40:16,006][SEVERE][grid-timeout-worker-#31][G] Blocked system-critical
thread has been detected. This can lead to cluster-wide undefined behaviour
[threadName=tcp-disco-msg-worker, blockedFor=119s]
[01:40:16,006][SEVERE][exchange-worker-#53][] Critical system error
detected. Will be handled accordingly toconfigured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=null, finished=true,heartbeatTs=1562030913742]]]class
org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=null, finished=true, heartbeatTs=1562030913742] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)
at java.lang.Thread.run(Thread.java:748)Exception in thread
"exchange-worker-#53" java.lang.OutOfMemoryError: Java heap
space[01:40:42,376][SEVERE][tcp-disco-msg-worker-#2][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.OutOfMemoryError: Java heap space]]java.lang.OutOfMemoryError:
Java heap space[01:40:47,316][SEVERE][tcp-disco-msg-worker-#2][] JVM will be
halted immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Durable Memory Runs out of Heap Space
Posted by Denis Magda <dm...@apache.org>.
Hi,
Ignite stores data off-heap (and that's what you've configured) while the
out-of-memory exception reports that Java Heap usage goes beyond some bar.
Please double check you have optimal heap settings and there are no memory
leaks:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
-
Denis
On Wed, Jul 3, 2019 at 2:55 PM jackluo923 <ja...@mail.utoronto.ca> wrote:
> Hi,
> I am trying to store many kv pairs into ignite than the available ram.
> At this moment, the primary purpose of using ignite is a distributed kv
> database.
>
> *My current configuration is: *
> <property name="defaultDataRegionConfiguration">
> <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
>
>
> <property name="persistenceEnabled" value="true"/>
>
>
> <property name="initialSize" value="#{500L * 1024 * 1024}"/>
>
>
> <property name="maxSize" value="#{32L * 1024 * 1024 * 1024}"/>
>
> </bean>
> </property>
>
> The physical machine has 64GB of ram. The data which I am trying to insert
> into ignite is spread across multiple caches (around 20). From my current
> understanding, we shouldn't receive any "Out of Heap Space" since the
> ignite
> uses "off heap storage" and anything which does not fit in the allocated
> cache space will be swapped to disk. Do I have any obvious flaw in my
> understanding or incorrect configuration?
>
> A small snippet of log message:
> [01:40:16,006][SEVERE][grid-timeout-worker-#31][G] Blocked system-critical
> thread has been detected. This can lead to cluster-wide undefined behaviour
> [threadName=tcp-disco-msg-worker, blockedFor=119s]
> [01:40:16,006][SEVERE][exchange-worker-#53][] Critical system error
> detected. Will be handled accordingly toconfigured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
> o.a.i.IgniteException: GridWorker [name=partition-exchanger,
> igniteInstanceName=null, finished=true,heartbeatTs=1562030913742]]]class
> org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger,
> igniteInstanceName=null, finished=true, heartbeatTs=1562030913742]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
>
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
>
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)
>
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)
>
> at java.lang.Thread.run(Thread.java:748)Exception in thread
> "exchange-worker-#53" java.lang.OutOfMemoryError: Java heap
> space[01:40:42,376][SEVERE][tcp-disco-msg-worker-#2][] Critical system
> error
> detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR,
> err=java.lang.OutOfMemoryError: Java heap
> space]]java.lang.OutOfMemoryError:
> Java heap space[01:40:47,316][SEVERE][tcp-disco-msg-worker-#2][] JVM will
> be
> halted immediately due to the failure: [failureCtx=FailureContext
> [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>