You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Josh Katz <Jo...@dodgeandcox.com> on 2021/03/10 16:53:46 UTC
Cluster keeps crashing with SEVERE message
After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,
After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.
Is there a way to salvage the data so we don't have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?
How can we avoid seeing this error? "[SEVERE] Failed to process selector key"
[cid:image001.png@01D7158A.E03AD0A0]
Thanks,
Josh
----------------------------------------------------------------------
Please follow the hyperlink to important disclosures.https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html
Re: Cluster keeps crashing with SEVERE message
Posted by Stephen Darlington <st...@gridgain.com>.
More heap-space. See the documentation for memory tuning: memory-tuning <https://ignite.apache.org/docs/latest/perf-and-troubleshooting/memory-tuning#generic-gc-settings>
You might also want to check that your query is iterating over the records in your cache rather than trying to read the whole thing into memory.
Regards,
Stephen
> On 10 Mar 2021, at 17:44, Josh Katz <Jo...@dodgeandcox.com> wrote:
>
> Please see the log, it seems to be OutOfMemory from JVM on Windows Server 2016.
> How can we address this issue?
>
> ^-- System thread pool [active=0, idle=7, qSize=0]
> [08:35:20,163][INFO][grid-timeout-worker-#22%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][IgniteKernal%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56] FreeList [name=default##FreeList, buckets=256, dataPages=2657, reusePages=0]
> [08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query [qry=GridCacheQueryInfo [loc=false, trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, filter=null, limit=0, transform=null, part=null, incMeta=false, pageSize=1024, timeout=0, incBackups=false, forceLocal=false, dedup=false, prj=null, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, mvccSnapshot=null, dataPageScanEnabled=null], locFut=null, sndId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, reqId=9, incMeta=false, all=false], node=10276c3c-a550-490e-bc48-edf9596dd1dd]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
> at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
> at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
> [08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query.
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
> at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
> at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
> [08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheIoManager] Failed processing message [senderId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, msg=GridCacheQueryRequest [id=9, cacheName=NonEquitiesYoY, type=SCAN, fields=false, clause=null, limit=0, clsName=null, keyValFilter=null, rdc=null, trans=null, pageSize=1024, incBackups=false, cancel=false, incMeta=false, all=false, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, part=-1, topVer=AffinityTopologyVersion [topVer=163, minorTopVer=0], mvccSnapshot=null, flags=0, super=GridCacheIdMessage [cacheId=-1867634189, super=GridCacheMessage [msgId=10, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=1], err=null, skipPrepare=false]]]]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
> at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
> at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
> [08:35:22,163][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
> at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
> at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
> at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
> at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
> at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
> [08:35:22,179][WARNING][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][CacheDiagnosticManager] Page locks dump:
>
> Thread=[name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=66], state=TIMED_WAITING
> Locked pages = []
> Locked pages log: name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=7400], state=RUNNABLE
> Locked pages = []
> Locked pages log: name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=11], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=12], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=13], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=14], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=15], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=16], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=17], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
> Thread=[name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=18], state=WAITING
> Locked pages = []
> Locked pages log: name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
>
>
>
> [08:35:22,179][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
>
>
>
> From: Stephen Darlington <stephen.darlington@gridgain.com <ma...@gridgain.com>>
> Sent: Wednesday, March 10, 2021 9:35 AM
> To: user@ignite.apache.org <ma...@ignite.apache.org>
> Subject: Re: Cluster keeps crashing with SEVERE message
>
> This is an EXTERNAL EMAIL. Stop and think before clicking a link or opening attachments.
>
> We’d need to see more of the stack trace and see your configuration before this could be diagnosed.
>
> It looks like a node abruptly left the cluster. If you’re losing data it seems like you’ve not configured any backups.
>
>
> On 10 Mar 2021, at 16:53, Josh Katz <Josh.Katz.Contractor@dodgeandcox.com <ma...@dodgeandcox.com>> wrote:
>
> After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,
> After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.
> Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?
> How can we avoid seeing this error? “[SEVERE] Failed to process selector key”
>
> <image001.png>
>
> Thanks,
> Josh
>
>
> Please follow the hyperlink to important disclosures.
> https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html <https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html>
RE: Cluster keeps crashing with SEVERE message
Posted by Josh Katz <Jo...@dodgeandcox.com>.
Please see the log, it seems to be OutOfMemory from JVM on Windows Server 2016.
How can we address this issue?
^-- System thread pool [active=0, idle=7, qSize=0]
[08:35:20,163][INFO][grid-timeout-worker-#22%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][IgniteKernal%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56] FreeList [name=default##FreeList, buckets=256, dataPages=2657, reusePages=0]
[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query [qry=GridCacheQueryInfo [loc=false, trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, filter=null, limit=0, transform=null, part=null, incMeta=false, pageSize=1024, timeout=0, incBackups=false, forceLocal=false, dedup=false, prj=null, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, mvccSnapshot=null, dataPageScanEnabled=null], locFut=null, sndId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, reqId=9, incMeta=false, all=false], node=10276c3c-a550-490e-bc48-edf9596dd1dd]
java.lang.OutOfMemoryError: Java heap space
at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] <NonEquitiesYoY> Failed to run query.
java.lang.OutOfMemoryError: Java heap space
at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheIoManager] Failed processing message [senderId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, msg=GridCacheQueryRequest [id=9, cacheName=NonEquitiesYoY, type=SCAN, fields=false, clause=null, limit=0, clsName=null, keyValFilter=null, rdc=null, trans=null, pageSize=1024, incBackups=false, cancel=false, incMeta=false, all=false, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, part=-1, topVer=AffinityTopologyVersion [topVer=163, minorTopVer=0], mvccSnapshot=null, flags=0, super=GridCacheIdMessage [cacheId=-1867634189, super=GridCacheMessage [msgId=10, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=1], err=null, skipPrepare=false]]]]
java.lang.OutOfMemoryError: Java heap space
at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:22,163][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
java.lang.OutOfMemoryError: Java heap space
at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160)
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131)
at org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164)
at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881)
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028)
at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314)
[08:35:22,179][WARNING][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][CacheDiagnosticManager] Page locks dump:
Thread=[name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=66], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=7400], state=RUNNABLE
Locked pages = []
Locked pages log: name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=11], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=12], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=13], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=14], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=15], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=16], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=17], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
Thread=[name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, id=18], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% time=(1615394122179, 2021-03-10 08:35:22.179)
[08:35:22,179][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
From: Stephen Darlington <st...@gridgain.com>
Sent: Wednesday, March 10, 2021 9:35 AM
To: user@ignite.apache.org
Subject: Re: Cluster keeps crashing with SEVERE message
This is an EXTERNAL EMAIL. Stop and think before clicking a link or opening attachments.
We’d need to see more of the stack trace and see your configuration before this could be diagnosed.
It looks like a node abruptly left the cluster. If you’re losing data it seems like you’ve not configured any backups.
On 10 Mar 2021, at 16:53, Josh Katz <Jo...@dodgeandcox.com>> wrote:
After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,
After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.
Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?
How can we avoid seeing this error? “[SEVERE] Failed to process selector key”
<image001.png>
Thanks,
Josh
________________________________
Please follow the hyperlink to important disclosures.
https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html
Re: Cluster keeps crashing with SEVERE message
Posted by Stephen Darlington <st...@gridgain.com>.
We’d need to see more of the stack trace and see your configuration before this could be diagnosed.
It looks like a node abruptly left the cluster. If you’re losing data it seems like you’ve not configured any backups.
> On 10 Mar 2021, at 16:53, Josh Katz <Jo...@dodgeandcox.com> wrote:
>
> After we load 35 million records from the database into the cluster. When we query it using a scan query we get the following SEVERE error,
> After that the node are restarting and the caches all show 0 keys and we have to restart the initial load.
> Is there a way to salvage the data so we don’t have to reload every time? For example, if we add persistence at this stage the initial load will be slower correct?
> How can we avoid seeing this error? “[SEVERE] Failed to process selector key”
>
> <image001.png>
>
> Thanks,
> Josh
>
>
> Please follow the hyperlink to important disclosures.
> https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html <https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html>