You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by consultant_dmitry <dm...@pwc.com> on 2017/12/01 06:45:06 UTC

Out of memory error

Hi Ignite team, 

My cluster is a windows server with 32 gb RAM (24 free). I built project in
gridgain.console and use default properties for my project (only change
Query parallelism parameter). When I run my project in IDEA I have next
error log:

[18:13:20] Ignite node started OK (id=7598c95e, instance name=Project)
[18:13:20] Topology snapshot [ver=2, servers=1, clients=1, CPUs=8,
heap=14.0GB]
>>> Loading caches...
>>> Loading cache: ProjectCache
[18:20:43,868][SEVERE][mgmt-#44%Project%][GridTaskWorker] Failed to obtain
remote job result policy for result from ComputeTask.result(..) method (will
fail the whole task): GridJobResultImpl [job=C2 [c=LoadCacheJobV2
[keepBinary=false]], sib=GridJobSiblingImpl
[sesId=fd92d7d0061-7598c95e-e614-49ff-bf47-eb59bf66003d,
jobId=0e92d7d0061-7598c95e-e614-49ff-bf47-eb59bf66003d,
nodeId=8f968d9c-10f7-4318-8ae1-271a9992e440, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=0e92d7d0061-7598c95e-e614-49ff-bf47-eb59bf66003d, timeoutObj=null,
attrs={}], node=TcpDiscoveryNode [id=8f968d9c-10f7-4318-8ae1-271a9992e440,
addrs=[0:0:0:0:0:0:0:1, 10.50.124.107, 127.0.0.1],
sockAddrs=[RU-LEDTEST07.cee.ema.ad.pwcinternal.com/10.50.124.107:47500,
/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1512054799051, loc=false,
ver=2.1.6#20171011-sha1:b6e23b35, isClient=false], ex=class
o.a.i.IgniteException: Failed to load cache: ProjectCache, hasRes=true,
isCancelled=false, isOccupied=true]
class org.apache.ignite.IgniteException: Remote job threw user exception
(override or implement ComputeTask.result(..) method if you would like to
have automatic failover for this exception).
	at
org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:101)
	at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1047)
	at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1040)
	at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6661)
	at
org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1040)
	at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:858)
	at
org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1066)
	at
org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1301)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1562)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1190)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
	at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Failed to load cache:
ProjectCache
	at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1858)
	at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:566)
	at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6629)
	at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:560)
	at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:489)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1115)
	at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1908)
	... 7 more
Caused by: class org.apache.ignite.IgniteException: Failed to load cache:
ProjectCache
	at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:966)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheJob.localExecute(GridCacheAdapter.java:5485)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheJobV2.localExecute(GridCacheAdapter.java:5529)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$TopologyVersionAwareJob.execute(GridCacheAdapter.java:6144)
	at
org.apache.ignite.compute.ComputeJobAdapter.call(ComputeJobAdapter.java:132)
	at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1855)
	... 14 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to load
cache: ProjectCache
	at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadCache(GridCacheStoreManagerAdapter.java:527)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.localLoadCache(GridDhtCacheAdapter.java:607)
	at
org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.localLoadCache(GridCacheProxyImpl.java:217)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheJob.localExecute(GridCacheAdapter.java:5480)
	... 18 more
Caused by: javax.cache.integration.CacheLoaderException: Failed to load
cache: ProjectCache
	at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.loadCache(CacheAbstractJdbcStore.java:819)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadCache(GridCacheStoreManagerAdapter.java:504)
	... 21 more
Caused by: class org.apache.ignite.IgniteException: Runtime failure on
search row:
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$SearchRow@240cee83
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1632)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1210)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3194)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2571)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.loadEntry(GridDhtCacheAdapter.java:659)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.access$600(GridDhtCacheAdapter.java:99)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$5.apply(GridDhtCacheAdapter.java:611)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$5.apply(GridDhtCacheAdapter.java:607)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter$3.apply(GridCacheStoreManagerAdapter.java:520)
	at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore$1.call(CacheAbstractJdbcStore.java:470)
	at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore$1.call(CacheAbstractJdbcStore.java:434)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
Caused by: class org.apache.ignite.internal.mem.IgniteOutOfMemoryException:
Not enough memory allocated (consider increasing memory policy size or
enabling evictions) [policyName=default, size=6.9 GB]
	at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:292)
	at
org.apache.ignite.internal.processors.cache.persistence.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:456)
	at
org.apache.ignite.internal.processors.cache.persistence.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:494)
	at
org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:84)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1264)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3890)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3848)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:2988)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2882)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1713)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1690)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1602)
	... 15 more
[18:20:44] Ignite node stopped OK [name=Project, uptime=00:07:23:779]
Exception in thread "main" javax.cache.integration.CacheLoaderException:
Failed to load cache: ProjectCache
	at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.loadCache(CacheAbstractJdbcStore.java:819)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadCache(GridCacheStoreManagerAdapter.java:504)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.localLoadCache(GridDhtCacheAdapter.java:607)
	at
org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.localLoadCache(GridCacheProxyImpl.java:217)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheJob.localExecute(GridCacheAdapter.java:5480)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$LoadCacheJobV2.localExecute(GridCacheAdapter.java:5529)
	at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$TopologyVersionAwareJob.execute(GridCacheAdapter.java:6144)
	at
org.apache.ignite.compute.ComputeJobAdapter.call(ComputeJobAdapter.java:132)
	at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1855)
	at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:566)
	at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6629)
	at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:560)
	at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:489)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1115)
	at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1908)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1562)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1190)
	at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
	at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Runtime failure on
search row:
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$SearchRow@240cee83
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1632)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1210)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3194)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2571)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.loadEntry(GridDhtCacheAdapter.java:659)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.access$600(GridDhtCacheAdapter.java:99)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$5.apply(GridDhtCacheAdapter.java:611)
	at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$5.apply(GridDhtCacheAdapter.java:607)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter$3.apply(GridCacheStoreManagerAdapter.java:520)
	at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore$1.call(CacheAbstractJdbcStore.java:470)
	at
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore$1.call(CacheAbstractJdbcStore.java:434)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
Caused by: class org.apache.ignite.internal.mem.IgniteOutOfMemoryException:
Not enough memory allocated (consider increasing memory policy size or
enabling evictions) [policyName=default, size=6.9 GB]
	at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:292)
	at
org.apache.ignite.internal.processors.cache.persistence.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:456)
	at
org.apache.ignite.internal.processors.cache.persistence.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:494)
	at
org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:84)
	at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1264)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3890)
	at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3848)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:2988)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2882)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1713)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1690)
	at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1602)
	... 15 more

Process finished with exit code 1
So I have a question: how many free RAM I need for example for 10 gb csv
file (15 gb table in PostgreSQL)?
What properties I need to change and where? I tried to change standart
properties adding this code in my project:
<property name="memoryConfiguration">
    <bean class="org.apache.ignite.configuration.MemoryConfiguration">
        <property name="memoryPolicies">
            <list>
                <bean
class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
                    <property name="initialSize" value="#{16L * 1024 * 1024
* 1024}"/>
                    <property name="maxSize" value="#{16L * 1024 * 1024 *
1024}"/>
                    <property name="metricsEnabled" value="true"/>
                </bean>
            </list>
        </property>
    </bean>
</property>

But I have the same error. 

Best regards, 
Kurgansky Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Out of memory error

Posted by Evgenii Zhuravlev <e....@gmail.com>.
Hi Dmitriy,

Looks like you configured memory policy and memory configuration that are
not used by your caches. Also, it looks like your cache use default memory
configuration, try to add <property name="defaultMemoryPolicyName" value=
"dfltPlc"/> to the memoryConfiguration and <property name="name" value=
"dfltPlc"/> to the memoryPolicy configuration

Evgenii

2017-12-01 16:42 GMT+03:00 Alexey Kukushkin <ku...@gmail.com>:

> Hi,
>
> You identified the problem right: there is not enough memory to handle 15G
> of Postgres data on your server. Your idea to configure a memory policy to
> increase available memory is right but 16G is also not enough. Ignite data
> size is noticeably larger (up to 3 times, depends on many factors like
> amount of indexes) than that of the relational DB data files on disk.
>
> You need to estimate how much memory you need. There are two options:
>
>    - Theoretical - see here
>    <https://apacheignite.readme.io/docs/capacity-planning>.
>    - Empirical: enable memory metrics
>    <https://apacheignite.readme.io/docs/memory-metrics> and load a
>    smaller amount of entries into cache. Use JMX client to check
>    "totalAllocatedPages" of the DataRegionMetricsMXBean on the server.
>    Multiply the number by 4K to get total used memory in kilobytes. Then
>    extrapolate to the max amount of entries you are going to have.
>
> Remember that Ignite saves you money by horizontal scaling. Just have 4
> 32G servers if you need 100G of RAM.
>
>
>
>

Re: Out of memory error

Posted by Alexey Kukushkin <ku...@gmail.com>.
Hi,

You identified the problem right: there is not enough memory to handle 15G
of Postgres data on your server. Your idea to configure a memory policy to
increase available memory is right but 16G is also not enough. Ignite data
size is noticeably larger (up to 3 times, depends on many factors like
amount of indexes) than that of the relational DB data files on disk.

You need to estimate how much memory you need. There are two options:

   - Theoretical - see here
   <https://apacheignite.readme.io/docs/capacity-planning>.
   - Empirical: enable memory metrics
   <https://apacheignite.readme.io/docs/memory-metrics> and load a smaller
   amount of entries into cache. Use JMX client to check "totalAllocatedPages"
   of the DataRegionMetricsMXBean on the server. Multiply the number by 4K to
   get total used memory in kilobytes. Then extrapolate to the max amount of
   entries you are going to have.

Remember that Ignite saves you money by horizontal scaling. Just have 4 32G
servers if you need 100G of RAM.