You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mahesh Renduchintala <ma...@aline-consulting.com> on 2020/11/19 17:15:45 UTC
Ignite 2.8.1: Database Closed error
Hi,
Any pointers on what the below error means.
There seems to be an out of memory on discoverySPI. what can cause this?
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=50071MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=50071MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=1, qSize=0]
[16:29:45,630][INFO][exchange-worker-#85][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=2016, minorTopVer=0], force=true, evt=DISCOVERY>
[16:30:12,877][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 2323 milliseconds.
[16:30:12,877][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/192.168.10.137, rmtPort=39599]
[16:30:12,882][SEVERE][tcp-disco-client-message-worker-[9a0c020b 192.168.1.9:61059]-#1560][TcpDiscoverySpi] Runtime error caught during grid runnable execution: GridWorker [name=tcp-disco-client-message>
java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7761)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[16:30:12,883][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/192.168.10.137, rmtPort=39599]
[16:30:12,883][SEVERE][query-#218652][GridMapQueryExecutor] Failed to execute local query.
class org.apache.ignite.IgniteCheckedException: Failed to execute SQL query. The database has been closed [90098-197]
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.engine.Database.checkPowerOff(Database.java:536)
at org.h2.command.Command.executeQuery(Command.java:228)
Re: Ignite 2.8.1: Database Closed error
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Hi,
I don't think it's related to the discovery. In the log, before OOM you can
see the long JVM pause:
Possible too long JVM pause: 2323 milliseconds.
So, probably you used more heap memory than you had.
What version do you use? How much heap memory do you have? What do you do
with a cluster(SQL, key value operations, etc.)? Also, can you share full
logs?
I would recommend collecting heap dump and checking objects that use memory
there.
Evgenii
чт, 19 нояб. 2020 г. в 09:16, Mahesh Renduchintala <
mahesh.renduchintala@aline-consulting.com>:
> Hi,
>
>
> Any pointers on what the below error means.
> There seems to be an out of memory on discoverySPI. what can cause this?
>
>
>
>
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=50071MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=50071MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=1, qSize=0]
> [16:29:45,630][INFO][exchange-worker-#85][GridCachePartitionExchangeManager]
> Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
> [topVer=2016, minorTopVer=0], force=true, evt=DISCOVERY>
> [16:30:12,877][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
> too long JVM pause: 2323 milliseconds.
> [16:30:12,877][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP
> discovery accepted incoming connection [rmtAddr=/192.168.10.137,
> rmtPort=39599]
> [16:30:12,882][SEVERE][tcp-disco-client-message-worker-[9a0c020b
> 192.168.1.9:61059]-#1560][TcpDiscoverySpi] Runtime error caught during
> grid runnable execution: GridWorker [name=tcp-disco-client-message>
> java.lang.OutOfMemoryError: Java heap space
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
> at
> java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
> at
> java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7761)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
> [16:30:12,883][INFO][tcp-disco-srvr-[:47500]-#3][TcpDiscoverySpi] TCP
> discovery spawning a new thread for connection [rmtAddr=/192.168.10.137,
> rmtPort=39599]
> [16:30:12,883][SEVERE][query-#218652][GridMapQueryExecutor] Failed to
> execute local query.
> class org.apache.ignite.IgniteCheckedException: Failed to execute SQL
> query. The database has been closed [90098-197]
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
> at
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
> at
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed
> [90098-197]
> at
> org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
> at org.h2.message.DbException.get(DbException.java:179)
> at org.h2.message.DbException.get(DbException.java:155)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.engine.Database.checkPowerOff(Database.java:536)
> at org.h2.command.Command.executeQuery(Command.java:228)
>
>
>
>