You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Vadim Dedkov <de...@gmail.com> on 2017/02/16 16:58:15 UTC
OutOfMemoryException
Hello to all.
I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7.
On my cluster I have 4 server nodes.
Each node have 8 cpu and 16gb memory.
GG visor info on each node: Heap memory initialized: 10gb, Heap memory
maximum: 10gb.
I tried to process file (~4.7gb) with Hadoop MR: parse and write file data
to GG cache.
And after a while since start job I got errors (end of letter).
I tried to analise error log & source code and one of my versions is: GG
tries to swap memory on hard disk --> I don't have enough free space in GG
directory --> error.
But I didn't setup cache config "swapEnabled" and I cannot find such config
option in Ignite 1.10.ea7 source code.
And in this hypothesis I cannot realise, why GG tries to use swap: 10gb *
4node is not enough for 4.7 gb data?
On one of my GG node:
[16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache]
<calcScorePointsCacheIn_2001> Unexpected exception during cache update
class org.apache.ignite.internal.mem.OutOfMemoryException: null
at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
at
org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
at
org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
at
org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:80)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:842)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:302)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1369)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:997)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:113)
at
org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:950)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
And in MR job log:
Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to
update keys (retry update if possible).: [00-1-10060940]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1478)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2118)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1315)
at
ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10)
at
ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36)
at
ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: class
org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException:
Failed to update keys (retry update if possible).: [00-1-10060940]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onResult(GridNearAtomicSingleUpdateFuture.java:227)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3077)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1369)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:997)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:113)
at
org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:950)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Suppressed: class org.apache.ignite.IgniteCheckedException: Failed
to update keys on primary node.
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:370)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
... 12 more
Suppressed: class
org.apache.ignite.internal.mem.OutOfMemoryException: null
at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
at
org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
at
org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
at
org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:80)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:842)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:302)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
... 17 more
--
_______________ _______________
Best regards, С уважением
Vadim Dedkov. Вадим Дедков.
Re: OutOfMemoryException
Posted by Andrey Mashenkov <am...@gridgain.com>.
Hi Vadim,
You wrote to Apache Ignite Community dev list.
Ignite does not have versions you mentioned.
On Thu, Feb 16, 2017 at 7:58 PM, Vadim Dedkov <de...@gmail.com> wrote:
> Hello to all.
> I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7.
> On my cluster I have 4 server nodes.
> Each node have 8 cpu and 16gb memory.
> GG visor info on each node: Heap memory initialized: 10gb, Heap memory
> maximum: 10gb.
> I tried to process file (~4.7gb) with Hadoop MR: parse and write file data
> to GG cache.
>
> And after a while since start job I got errors (end of letter).
> I tried to analise error log & source code and one of my versions is: GG
> tries to swap memory on hard disk --> I don't have enough free space in GG
> directory --> error.
> But I didn't setup cache config "swapEnabled" and I cannot find such
> config option in Ignite 1.10.ea7 source code.
> And in this hypothesis I cannot realise, why GG tries to use swap: 10gb *
> 4node is not enough for 4.7 gb data?
>
>
> On one of my GG node:
> [16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache]
> <calcScorePointsCacheIn_2001> Unexpected exception during cache update
> class org.apache.ignite.internal.mem.OutOfMemoryException: null
> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:242)
> at org.apache.ignite.internal.processors.cache.database.
> freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
> at org.apache.ignite.internal.processors.cache.database.
> freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
> at org.apache.ignite.internal.processors.cache.database.
> RowStore.addRow(RowStore.java:80)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(
> IgniteCacheOffheapManagerImpl.java:842)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.
> java:302)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(
> GridDhtAtomicCache.java:1755)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(
> GridDhtAtomicCache.java:1593)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest
> (GridDhtAtomicCache.java:3059)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> processMessage(GridCacheIoManager.java:758)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> handleMessage(GridCacheIoManager.java:287)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$000(GridCacheIoManager.java:89)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1369)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:997)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:113)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager$6.run(GridIoManager.java:950)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> And in MR job log:
> Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to
> update keys (retry update if possible).: [00-1-10060940]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1478)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2118)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1315)
> at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10)
> at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36)
> at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(
> ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: class org.apache.ignite.internal.processors.cache.
> CachePartialUpdateCheckedException: Failed to update keys (retry update
> if possible).: [00-1-10060940]
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridNearAtomicSingleUpdateFuture.onResult(
> GridNearAtomicSingleUpdateFuture.java:227)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRespons
> e(GridDhtAtomicCache.java:3077)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$000(GridCacheIoManager.java:89)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1369)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:997)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:113)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager$6.run(GridIoManager.java:950)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: class org.apache.ignite.IgniteCheckedException:
> Failed to update keys on primary node.
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.
> addFailedKeys(GridNearAtomicUpdateResponse.java:370)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> access$600(GridDhtAtomicCache.java:131)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:288)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:283)
> ... 12 more
> Suppressed: class org.apache.ignite.internal.mem.OutOfMemoryException:
> null
> at org.apache.ignite.internal.pagemem.impl.
> PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
> at org.apache.ignite.internal.
> processors.cache.database.freelist.FreeListImpl.
> allocateDataPage(FreeListImpl.java:295)
> at org.apache.ignite.internal.
> processors.cache.database.freelist.FreeListImpl.
> insertDataRow(FreeListImpl.java:325)
> at org.apache.ignite.internal.
> processors.cache.database.RowStore.addRow(RowStore.java:80)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(
> IgniteCacheOffheapManagerImpl.java:842)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.
> java:302)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(
> GridDhtAtomicCache.java:2353)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
> ... 17 more
>
>
>
> --
> _______________ _______________
> Best regards, С уважением
> Vadim Dedkov. Вадим Дедков.
>
Re: OutOfMemoryException
Posted by Vadim Dedkov <de...@gmail.com>.
I wrote to Aleksey Shcherbakov and he advised me to set parameter
"GridDatabaseConfiguration.pageCacheSize = 53687091200" (50 GB).
After that I have changed my GG config like this (be careful, it was not
GridDatabaseConfiguration-bean exactly):
<beans xmlns="http://www.springframework.org/schema/beans" ... >
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="clientMode" value="false"/>
...
<property name="memoryConfiguration">
<bean
class="org.apache.ignite.configuration.MemoryConfiguration">
<property name="pageCacheSize" value="53687091200" />
</bean>
</property>
</bean>
</beans>
This helped me to overcome OutOfMemoryException.
2017-02-17 6:04 GMT+03:00 Denis Magda <dm...@gridgain.com>:
> Hello Vadim,
>
> This is a support channel of Apache Ignite community. You’ll not get
> support for GridGain here.
>
> Please talk to your colleagues or Alexey Scherbakov (cc-ed) asking on how
> get access to FreshDesk which is GridGain’s support portal.
>
>
>
> Regards,
> --
> Denis Magda
> Product Manager, GridGain Systems <https://www.gridgain.com>
>
> Begin forwarded message:
>
> *From: *Vadim Dedkov <de...@gmail.com>
> *Subject: **OutOfMemoryException*
> *Date: *February 16, 2017 at 8:58:15 AM PST
> *To: *user@ignite.apache.org
> *Reply-To: *user@ignite.apache.org
>
> Hello to all.
> I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7.
> On my cluster I have 4 server nodes.
> Each node have 8 cpu and 16gb memory.
> GG visor info on each node: Heap memory initialized: 10gb, Heap memory
> maximum: 10gb.
> I tried to process file (~4.7gb) with Hadoop MR: parse and write file data
> to GG cache.
>
> And after a while since start job I got errors (end of letter).
> I tried to analise error log & source code and one of my versions is: GG
> tries to swap memory on hard disk --> I don't have enough free space in GG
> directory --> error.
> But I didn't setup cache config "swapEnabled" and I cannot find such
> config option in Ignite 1.10.ea7 source code.
> And in this hypothesis I cannot realise, why GG tries to use swap: 10gb *
> 4node is not enough for 4.7 gb data?
>
>
> On one of my GG node:
> [16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache]
> <calcScorePointsCacheIn_2001> Unexpected exception during cache update
> class org.apache.ignite.internal.mem.OutOfMemoryException: null
> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:242)
> at org.apache.ignite.internal.processors.cache.database.
> freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
> at org.apache.ignite.internal.processors.cache.database.
> freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
> at org.apache.ignite.internal.processors.cache.database.
> RowStore.addRow(RowStore.java:80)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(
> IgniteCacheOffheapManagerImpl.java:842)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.
> java:302)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(
> GridDhtAtomicCache.java:1755)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(
> GridDhtAtomicCache.java:1593)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest
> (GridDhtAtomicCache.java:3059)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> processMessage(GridCacheIoManager.java:758)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> handleMessage(GridCacheIoManager.java:287)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$000(GridCacheIoManager.java:89)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1369)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:997)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:113)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager$6.run(GridIoManager.java:950)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> And in MR job log:
> Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to
> update keys (retry update if possible).: [00-1-10060940]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1478)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2118)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1315)
> at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10)
> at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36)
> at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(
> ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: class org.apache.ignite.internal.processors.cache.
> CachePartialUpdateCheckedException: Failed to update keys (retry update
> if possible).: [00-1-10060940]
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridNearAtomicSingleUpdateFuture.onResult(
> GridNearAtomicSingleUpdateFuture.java:227)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRespons
> e(GridDhtAtomicCache.java:3077)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$000(GridCacheIoManager.java:89)
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1369)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:997)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:113)
> at org.apache.ignite.internal.managers.communication.
> GridIoManager$6.run(GridIoManager.java:950)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: class org.apache.ignite.IgniteCheckedException:
> Failed to update keys on primary node.
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.
> addFailedKeys(GridNearAtomicUpdateResponse.java:370)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> access$600(GridDhtAtomicCache.java:131)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:288)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:283)
> ... 12 more
> Suppressed: class org.apache.ignite.internal.mem.OutOfMemoryException:
> null
> at org.apache.ignite.internal.pagemem.impl.
> PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
> at org.apache.ignite.internal.
> processors.cache.database.freelist.FreeListImpl.
> allocateDataPage(FreeListImpl.java:295)
> at org.apache.ignite.internal.
> processors.cache.database.freelist.FreeListImpl.
> insertDataRow(FreeListImpl.java:325)
> at org.apache.ignite.internal.
> processors.cache.database.RowStore.addRow(RowStore.java:80)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(
> IgniteCacheOffheapManagerImpl.java:842)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.
> java:302)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
> at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(
> GridDhtAtomicCache.java:2353)
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
> ... 17 more
>
>
>
> --
> _______________ _______________
> Best regards, С уважением
> Vadim Dedkov. Вадим Дедков.
>
>
>
--
_______________ _______________
Best regards, С уважением
Vadim Dedkov. Вадим Дедков.