You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by DonTequila <do...@gmx.de> on 2021/02/23 19:06:37 UTC
Database has been closed
I've a production cluster of 1 server with Ignite native persistence enabled,
and about 25 clients. For second time now I've seen the server node to fail
and stop which brings the whole cluster down. The collected log from the
server is with the link below.
ignite-2dc05669.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2401/ignite-2dc05669.log>
Some (to me at least) interesting parts are close to the end in the log.
What can cause such issue? Network issue? How can these be better configured
to be more resistant? Or slow HDD disk? Is there a configuration that can
prevent the whole cluster to go down?
I've left the Ignite communications at default values.
[05:48:11,677][WARNING][sys-stripe-26-#27][GridContinuousProcessor] Failed
to wait for ack message. [node=98fc4372-044b-4f9d-bd97-6f8237a0435b,
routine=95e1b30d-c37f-4daf-a605-eabdf08332e7]
[05:48:40,694][WARNING][sys-stripe-109-#110][GridContinuousProcessor] Failed
to wait for ack message. [node=e8e8026b-d3a6-4572-89ce-3c8777f7373f,
routine=b92d00bd-2006-4fc7-a8a3-e3f3c10e1f53]
[05:49:01,166][WARNING][sys-stripe-69-#70][GridContinuousProcessor] Failed
to wait for ack message. [node=5dea8244-0614-46db-b4e1-d5ccfd0be60b,
routine=b0c0f754-f9c0-4285-a6ad-863f92c49a06]
[05:52:26,332][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=ce957e46-98a6-4817-b73e-b604bf4548fd,
routine=6f455c23-9bce-4050-98ce-a1b45c0bad76]
[05:52:27,832][WARNING][sys-stripe-112-#113][GridContinuousProcessor] Failed
to wait for ack message. [node=f61f15ee-9a7e-41fc-8f2c-1db7d0d2d274,
routine=f6cddd41-130c-4f6f-b09b-f87942755bab]
[05:52:28,315][WARNING][sys-stripe-62-#63][GridContinuousProcessor] Failed
to wait for ack message. [node=aa9b1426-f4e0-47eb-a730-c67c8186e0a3,
routine=3a8db1ad-e201-427b-8b82-2ca7170ebf99]
[05:52:33,582][WARNING][sys-stripe-50-#51][GridContinuousProcessor] Failed
to wait for ack message. [node=907e5822-a6dc-48ce-bf44-892d92a7a1d8,
routine=d2ed54e7-528f-4b5c-8508-24333fc2c422]
[05:52:33,889][WARNING][sys-stripe-50-#51][GridContinuousProcessor] Failed
to wait for ack message. [node=117776ce-fd40-4523-9578-58605b2256be,
routine=20e30155-5cdc-4fae-b60e-1296cc4f2535]
[05:52:34,177][WARNING][sys-stripe-46-#47][GridContinuousProcessor] Failed
to wait for ack message. [node=dc614d74-98a7-4a82-8424-d7a147374d78,
routine=738a6030-f94d-4493-9fb2-cf8a5eb91439]
[05:52:34,682][WARNING][sys-stripe-121-#122][GridContinuousProcessor] Failed
to wait for ack message. [node=f284c2b3-e4c5-4e55-bd8b-5af63378f3c1,
routine=6d10df57-b152-47ee-8ac7-e9157b6c30ce]
[05:52:35,173][WARNING][sys-stripe-121-#122][GridContinuousProcessor] Failed
to wait for ack message. [node=3996b4ba-5850-42a3-a56b-0e2b3434c329,
routine=8056702f-262b-4f2d-ac6a-55c5ceb88f19]
[05:52:35,413][WARNING][sys-stripe-121-#122][GridContinuousProcessor] Failed
to wait for ack message. [node=51b85275-a775-4b36-b102-123c4a69fbd9,
routine=13d57b4a-9e36-48a8-80ae-a2afbcb5aa4e]
[05:52:36,128][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=29735637-620b-4ab7-a384-67499f98b1ef,
routine=c219cebb-17ae-4f88-bddb-62a24d73b186]
[05:52:36,381][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=b8d0ec72-63ff-4b6e-b232-d94d9d041fa1,
routine=34afbecb-7e3a-4fda-bcf7-95ee01765165]
[05:52:36,381][WARNING][long-qry-#221][LongRunningQueryManager] Query
execution is too long [time=3164ms, type=MAP, distributedJoin=false,
enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
node=TcpDiscoveryNode [id=6e0fba7d-c9a0-4c27-9492-2e231eeb3493,
consistentId=6e0fba7d-c9a0-4c27-9492-2e231eeb3493, addrs=ArrayList
[110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
[/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0], discPort=0,
order=351, intOrder=201, lastExchangeTime=1613877831620, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=39820, segment=0,
sql='SELECT
"MediaFan_TaskList".__Z0._KEY __C0_0,
"MediaFan_TaskList".__Z0._VAL __C0_1,
__Z0.CREATED __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC LIMIT 200', plan=SELECT
"MediaFan_TaskList".__Z0._KEY AS __C0_0,
"MediaFan_TaskList".__Z0._VAL AS __C0_1,
__Z0.CREATED AS __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
/* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
/* scanCount: 77490 */
WHERE (__Z0.STATUS = ?9)
OR ((__Z0.STATUS = ?8)
OR ((__Z0.STATUS = ?7)
OR ((__Z0.STATUS = ?6)
OR ((__Z0.STATUS = ?5)
OR ((__Z0.STATUS = ?4)
OR ((__Z0.STATUS = ?3)
OR ((__Z0.STATUS = ?1)
OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC
LIMIT 200]
[05:52:36,642][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=f69d6152-4f89-42cb-ad46-276124215f3b,
routine=d3fda97f-eb9a-403c-9af2-2cdccc6d2e88]
[05:52:37,118][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=eb5b51db-1847-40d6-b162-e1fb78e1ed85,
routine=40f34cf2-07cd-474c-b832-09332c7d4bc8]
[05:52:37,597][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=cc9423c8-503d-4964-b59d-c1223f07068e,
routine=1e97f6e0-cf42-4102-922f-0bd59507cdfd]
[05:52:37,826][WARNING][long-qry-#221][LongRunningQueryManager] Query
execution is too long [time=4004ms, type=MAP, distributedJoin=false,
enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
node=TcpDiscoveryNode [id=92945d9e-387c-467c-a47b-9f3be5b41574,
consistentId=92945d9e-387c-467c-a47b-9f3be5b41574, addrs=ArrayList
[110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
[/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0], discPort=0,
order=361, intOrder=206, lastExchangeTime=1613978781915, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=14040, segment=0,
sql='SELECT
"MediaFan_TaskList".__Z0._KEY __C0_0,
"MediaFan_TaskList".__Z0._VAL __C0_1,
__Z0.CREATED __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC LIMIT 200', plan=SELECT
"MediaFan_TaskList".__Z0._KEY AS __C0_0,
"MediaFan_TaskList".__Z0._VAL AS __C0_1,
__Z0.CREATED AS __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
/* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
/* scanCount: 45538 */
WHERE (__Z0.STATUS = ?9)
OR ((__Z0.STATUS = ?8)
OR ((__Z0.STATUS = ?7)
OR ((__Z0.STATUS = ?6)
OR ((__Z0.STATUS = ?5)
OR ((__Z0.STATUS = ?4)
OR ((__Z0.STATUS = ?3)
OR ((__Z0.STATUS = ?1)
OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC
LIMIT 200]
[05:52:38,036][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
to wait for ack message. [node=affbdc38-65e0-49f0-9a82-05f1611a95fb,
routine=35dffec7-068f-4b5c-8e57-2b6f08c0bb04]
[05:52:38,036][WARNING][long-qry-#221][LongRunningQueryManager] Query
execution is too long [time=4582ms, type=MAP, distributedJoin=false,
enforceJoinOrder=true, lazy=false, schema=MediaFan_BladeServices,
node=TcpDiscoveryNode [id=d9a00801-ec3b-4b38-bbba-7999b5cf428c,
consistentId=d9a00801-ec3b-4b38-bbba-7999b5cf428c, addrs=ArrayList
[110.10.123.101], sockAddrs=HashSet [/110.10.123.101:0], discPort=0,
order=428, intOrder=226, lastExchangeTime=1613988731051, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=256671, segment=0,
sql='SELECT
"MediaFan_BladeServices".__Z0._KEY __C0_0,
"MediaFan_BladeServices".__Z0._VAL __C0_1
FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
WHERE (__Z0.TEXTVALUE = ?3) AND ((__Z0.MODULENAME = ?1) AND (__Z0.NAME =
?2))', plan=SELECT
"MediaFan_BladeServices".__Z0._KEY AS __C0_0,
"MediaFan_BladeServices".__Z0._VAL AS __C0_1
FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
/* "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES_NAME_ASC_IDX: NAME =
?2 */
/* scanCount: 95044 */
WHERE (__Z0.TEXTVALUE = ?3)
AND ((__Z0.MODULENAME = ?1)
AND (__Z0.NAME = ?2))]
[05:52:38,490][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=74e98ce5-3159-4fb6-b5b0-219717460781,
routine=012d85e2-1e68-429f-bab9-7c33b41ea0ed]
[05:52:39,162][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=ebc3cde0-72e4-4941-9a2d-855a5ec6c260,
routine=41ce659a-a56d-4253-820a-ffdd026e4351]
[05:52:40,292][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=86331a2a-8486-47d7-8cbc-38a08348d7ce,
routine=48925ed1-34a9-4e9a-8bfd-55759729f712]
[05:52:40,553][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=d194f864-b8bd-45df-bf04-8d3c2e5f4bac,
routine=ba749835-ecef-492e-902e-8a606f624ae8]
[05:52:40,817][WARNING][long-qry-#221][LongRunningQueryManager] Query
execution is too long [time=6476ms, type=MAP, distributedJoin=false,
enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
node=TcpDiscoveryNode [id=92945d9e-387c-467c-a47b-9f3be5b41574,
consistentId=92945d9e-387c-467c-a47b-9f3be5b41574, addrs=ArrayList
[110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
[/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0], discPort=0,
order=361, intOrder=206, lastExchangeTime=1613978781915, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=14040, segment=0,
sql='SELECT
"MediaFan_TaskList".__Z0._KEY __C0_0,
"MediaFan_TaskList".__Z0._VAL __C0_1,
__Z0.CREATED __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC LIMIT 200', plan=SELECT
"MediaFan_TaskList".__Z0._KEY AS __C0_0,
"MediaFan_TaskList".__Z0._VAL AS __C0_1,
__Z0.CREATED AS __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
/* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
/* scanCount: 48739 */
WHERE (__Z0.STATUS = ?9)
OR ((__Z0.STATUS = ?8)
OR ((__Z0.STATUS = ?7)
OR ((__Z0.STATUS = ?6)
OR ((__Z0.STATUS = ?5)
OR ((__Z0.STATUS = ?4)
OR ((__Z0.STATUS = ?3)
OR ((__Z0.STATUS = ?1)
OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC
LIMIT 200]
[05:52:40,817][WARNING][long-qry-#221][LongRunningQueryManager] Query
execution is too long [time=7574ms, type=MAP, distributedJoin=false,
enforceJoinOrder=true, lazy=false, schema=MediaFan_BladeServices,
node=TcpDiscoveryNode [id=d9a00801-ec3b-4b38-bbba-7999b5cf428c,
consistentId=d9a00801-ec3b-4b38-bbba-7999b5cf428c, addrs=ArrayList
[110.10.123.101], sockAddrs=HashSet [/110.10.123.101:0], discPort=0,
order=428, intOrder=226, lastExchangeTime=1613988731051, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=256671, segment=0,
sql='SELECT
"MediaFan_BladeServices".__Z0._KEY __C0_0,
"MediaFan_BladeServices".__Z0._VAL __C0_1
FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
WHERE (__Z0.TEXTVALUE = ?3) AND ((__Z0.MODULENAME = ?1) AND (__Z0.NAME =
?2))', plan=SELECT
"MediaFan_BladeServices".__Z0._KEY AS __C0_0,
"MediaFan_BladeServices".__Z0._VAL AS __C0_1
FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
/* "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES_NAME_ASC_IDX: NAME =
?2 */
/* scanCount: 98066 */
WHERE (__Z0.TEXTVALUE = ?3)
AND ((__Z0.MODULENAME = ?1)
AND (__Z0.NAME = ?2))]
[05:52:40,817][WARNING][long-qry-#221][LongRunningQueryManager] Query
execution is too long [time=7853ms, type=MAP, distributedJoin=false,
enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
node=TcpDiscoveryNode [id=6e0fba7d-c9a0-4c27-9492-2e231eeb3493,
consistentId=6e0fba7d-c9a0-4c27-9492-2e231eeb3493, addrs=ArrayList
[110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
[/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0], discPort=0,
order=351, intOrder=201, lastExchangeTime=1613877831620, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=39820, segment=0,
sql='SELECT
"MediaFan_TaskList".__Z0._KEY __C0_0,
"MediaFan_TaskList".__Z0._VAL __C0_1,
__Z0.CREATED __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC LIMIT 200', plan=SELECT
"MediaFan_TaskList".__Z0._KEY AS __C0_0,
"MediaFan_TaskList".__Z0._VAL AS __C0_1,
__Z0.CREATED AS __C0_2
FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
/* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
/* scanCount: 83014 */
WHERE (__Z0.STATUS = ?9)
OR ((__Z0.STATUS = ?8)
OR ((__Z0.STATUS = ?7)
OR ((__Z0.STATUS = ?6)
OR ((__Z0.STATUS = ?5)
OR ((__Z0.STATUS = ?4)
OR ((__Z0.STATUS = ?3)
OR ((__Z0.STATUS = ?1)
OR (__Z0.STATUS = ?2))))))))
ORDER BY 3 DESC
LIMIT 200]
[05:52:41,057][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=619f1567-5a56-4473-8fba-83ee7a82ea07,
routine=da2db058-b375-4713-8bf6-e788ca315291]
[05:52:42,281][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=99853e49-dbe6-4641-aa91-809a78c76585,
routine=4d5d484d-4ec7-4511-9ac8-5e629df6ad09]
[05:52:51,886][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
too long JVM pause: 2499 milliseconds.
[05:52:51,888][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
to wait for ack message. [node=e09111c4-8826-4a6a-a9b9-6e21c519e055,
routine=6b66215d-4c72-4be3-8deb-51a91ad8bbe4]
[05:52:51,890][SEVERE][query-#6221][GridMapQueryExecutor] Failed to execute
local query.
class org.apache.ignite.IgniteCheckedException: Failed to execute SQL query.
The database has been closed [90098-197]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
at
org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed
[90098-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.engine.Database.checkPowerOff(Database.java:536)
at org.h2.command.Command.executeQuery(Command.java:228)
at
org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:867)
... 13 more
[05:52:51,899][SEVERE][query-#6186][GridMapQueryExecutor] Failed to execute
local query.
class org.apache.ignite.IgniteCheckedException: Failed to execute SQL query.
The database has been closed [90098-197]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
at
org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed
[90098-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.engine.Database.checkPowerOff(Database.java:536)
at org.h2.command.Command.executeQuery(Command.java:228)
at
org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:867)
... 13 more
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Database has been closed
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You don't seem to have INFO log level enabled, so there's not much to see
there unfortunately. Please enable INFO level, it should also add thread
dumps automatically, I think.
Regards,
--
Ilya Kasnacheev
пн, 1 мар. 2021 г. в 13:48, DonTequila <do...@gmx.de>:
> Hi Ilya,
>
> The complete log is with the link below, can you take a look:
> ignite-2dc05669.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2401/ignite-2dc05669.gz>
>
>
> I will try to add code that dumps stacktrace and threads in such cases, but
> I'm not sure yet how to do this.
>
> Thanks,
> Thomas.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Database has been closed
Posted by DonTequila <do...@gmx.de>.
Hi Ilya,
The complete log is with the link below, can you take a look:
ignite-2dc05669.gz
<http://apache-ignite-users.70518.x6.nabble.com/file/t2401/ignite-2dc05669.gz>
I will try to add code that dumps stacktrace and threads in such cases, but
I'm not sure yet how to do this.
Thanks,
Thomas.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Database has been closed
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Unfortunately, we can't go from here without thread dump or, at least, a
complete log from the beginning.
Regards,
--
Ilya Kasnacheev
чт, 25 февр. 2021 г. в 11:07, DonTequila <do...@gmx.de>:
> Hi again,
>
> is there anything that could help me identify the potential reason for the
> read lock timout without having a thread dump? Are there settings that I
> can
> tune to increase this timeout and prevent from the node/database to close?
>
> Thanks.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Database has been closed
Posted by DonTequila <do...@gmx.de>.
Hi again,
is there anything that could help me identify the potential reason for the
read lock timout without having a thread dump? Are there settings that I can
tune to increase this timeout and prevent from the node/database to close?
Thanks.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Database has been closed
Posted by DonTequila <do...@gmx.de>.
Hi,
thank you for your response. Yes I realized the read lock acquisition
timeout as well.
Unfortunately I don't have a thread dump for the situation when this happens
as the app just stops and I have no chance to get a thread dump. Do you have
a suggestion how to get a thread dump at this type of failure?
Also, what could cause read lock acquisition to time out? Slow disk?
Regards,
Thomas.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Database has been closed
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
The interesting parts are at the very beginning:
Checkpoint read lock acquisition has been timed out.
In this case you need to collect full thread dump from the node to check
which thread is not releasing checkpoint lock, and why.
This log does not contain thread dumps, so it seems.
You can also see
java.lang.OutOfMemoryError: Java heap space
But that's probably due to build-up of requests, etc, caused by the locking.
Regards,
--
Ilya Kasnacheev
вт, 23 февр. 2021 г. в 22:06, DonTequila <do...@gmx.de>:
> I've a production cluster of 1 server with Ignite native persistence
> enabled,
> and about 25 clients. For second time now I've seen the server node to fail
> and stop which brings the whole cluster down. The collected log from the
> server is with the link below.
>
> ignite-2dc05669.log
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2401/ignite-2dc05669.log>
>
>
> Some (to me at least) interesting parts are close to the end in the log.
> What can cause such issue? Network issue? How can these be better
> configured
> to be more resistant? Or slow HDD disk? Is there a configuration that can
> prevent the whole cluster to go down?
>
> I've left the Ignite communications at default values.
>
> [05:48:11,677][WARNING][sys-stripe-26-#27][GridContinuousProcessor] Failed
> to wait for ack message. [node=98fc4372-044b-4f9d-bd97-6f8237a0435b,
> routine=95e1b30d-c37f-4daf-a605-eabdf08332e7]
> [05:48:40,694][WARNING][sys-stripe-109-#110][GridContinuousProcessor]
> Failed
> to wait for ack message. [node=e8e8026b-d3a6-4572-89ce-3c8777f7373f,
> routine=b92d00bd-2006-4fc7-a8a3-e3f3c10e1f53]
> [05:49:01,166][WARNING][sys-stripe-69-#70][GridContinuousProcessor] Failed
> to wait for ack message. [node=5dea8244-0614-46db-b4e1-d5ccfd0be60b,
> routine=b0c0f754-f9c0-4285-a6ad-863f92c49a06]
> [05:52:26,332][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=ce957e46-98a6-4817-b73e-b604bf4548fd,
> routine=6f455c23-9bce-4050-98ce-a1b45c0bad76]
> [05:52:27,832][WARNING][sys-stripe-112-#113][GridContinuousProcessor]
> Failed
> to wait for ack message. [node=f61f15ee-9a7e-41fc-8f2c-1db7d0d2d274,
> routine=f6cddd41-130c-4f6f-b09b-f87942755bab]
> [05:52:28,315][WARNING][sys-stripe-62-#63][GridContinuousProcessor] Failed
> to wait for ack message. [node=aa9b1426-f4e0-47eb-a730-c67c8186e0a3,
> routine=3a8db1ad-e201-427b-8b82-2ca7170ebf99]
> [05:52:33,582][WARNING][sys-stripe-50-#51][GridContinuousProcessor] Failed
> to wait for ack message. [node=907e5822-a6dc-48ce-bf44-892d92a7a1d8,
> routine=d2ed54e7-528f-4b5c-8508-24333fc2c422]
> [05:52:33,889][WARNING][sys-stripe-50-#51][GridContinuousProcessor] Failed
> to wait for ack message. [node=117776ce-fd40-4523-9578-58605b2256be,
> routine=20e30155-5cdc-4fae-b60e-1296cc4f2535]
> [05:52:34,177][WARNING][sys-stripe-46-#47][GridContinuousProcessor] Failed
> to wait for ack message. [node=dc614d74-98a7-4a82-8424-d7a147374d78,
> routine=738a6030-f94d-4493-9fb2-cf8a5eb91439]
> [05:52:34,682][WARNING][sys-stripe-121-#122][GridContinuousProcessor]
> Failed
> to wait for ack message. [node=f284c2b3-e4c5-4e55-bd8b-5af63378f3c1,
> routine=6d10df57-b152-47ee-8ac7-e9157b6c30ce]
> [05:52:35,173][WARNING][sys-stripe-121-#122][GridContinuousProcessor]
> Failed
> to wait for ack message. [node=3996b4ba-5850-42a3-a56b-0e2b3434c329,
> routine=8056702f-262b-4f2d-ac6a-55c5ceb88f19]
> [05:52:35,413][WARNING][sys-stripe-121-#122][GridContinuousProcessor]
> Failed
> to wait for ack message. [node=51b85275-a775-4b36-b102-123c4a69fbd9,
> routine=13d57b4a-9e36-48a8-80ae-a2afbcb5aa4e]
> [05:52:36,128][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=29735637-620b-4ab7-a384-67499f98b1ef,
> routine=c219cebb-17ae-4f88-bddb-62a24d73b186]
> [05:52:36,381][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=b8d0ec72-63ff-4b6e-b232-d94d9d041fa1,
> routine=34afbecb-7e3a-4fda-bcf7-95ee01765165]
> [05:52:36,381][WARNING][long-qry-#221][LongRunningQueryManager] Query
> execution is too long [time=3164ms, type=MAP, distributedJoin=false,
> enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
> node=TcpDiscoveryNode [id=6e0fba7d-c9a0-4c27-9492-2e231eeb3493,
> consistentId=6e0fba7d-c9a0-4c27-9492-2e231eeb3493, addrs=ArrayList
> [110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
> [/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0],
> discPort=0,
> order=351, intOrder=201, lastExchangeTime=1613877831620, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=39820, segment=0,
> sql='SELECT
> "MediaFan_TaskList".__Z0._KEY __C0_0,
> "MediaFan_TaskList".__Z0._VAL __C0_1,
> __Z0.CREATED __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
> ((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
> ((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC LIMIT 200', plan=SELECT
> "MediaFan_TaskList".__Z0._KEY AS __C0_0,
> "MediaFan_TaskList".__Z0._VAL AS __C0_1,
> __Z0.CREATED AS __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> /* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
> /* scanCount: 77490 */
> WHERE (__Z0.STATUS = ?9)
> OR ((__Z0.STATUS = ?8)
> OR ((__Z0.STATUS = ?7)
> OR ((__Z0.STATUS = ?6)
> OR ((__Z0.STATUS = ?5)
> OR ((__Z0.STATUS = ?4)
> OR ((__Z0.STATUS = ?3)
> OR ((__Z0.STATUS = ?1)
> OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC
> LIMIT 200]
> [05:52:36,642][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=f69d6152-4f89-42cb-ad46-276124215f3b,
> routine=d3fda97f-eb9a-403c-9af2-2cdccc6d2e88]
> [05:52:37,118][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=eb5b51db-1847-40d6-b162-e1fb78e1ed85,
> routine=40f34cf2-07cd-474c-b832-09332c7d4bc8]
> [05:52:37,597][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=cc9423c8-503d-4964-b59d-c1223f07068e,
> routine=1e97f6e0-cf42-4102-922f-0bd59507cdfd]
> [05:52:37,826][WARNING][long-qry-#221][LongRunningQueryManager] Query
> execution is too long [time=4004ms, type=MAP, distributedJoin=false,
> enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
> node=TcpDiscoveryNode [id=92945d9e-387c-467c-a47b-9f3be5b41574,
> consistentId=92945d9e-387c-467c-a47b-9f3be5b41574, addrs=ArrayList
> [110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
> [/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0],
> discPort=0,
> order=361, intOrder=206, lastExchangeTime=1613978781915, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=14040, segment=0,
> sql='SELECT
> "MediaFan_TaskList".__Z0._KEY __C0_0,
> "MediaFan_TaskList".__Z0._VAL __C0_1,
> __Z0.CREATED __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
> ((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
> ((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC LIMIT 200', plan=SELECT
> "MediaFan_TaskList".__Z0._KEY AS __C0_0,
> "MediaFan_TaskList".__Z0._VAL AS __C0_1,
> __Z0.CREATED AS __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> /* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
> /* scanCount: 45538 */
> WHERE (__Z0.STATUS = ?9)
> OR ((__Z0.STATUS = ?8)
> OR ((__Z0.STATUS = ?7)
> OR ((__Z0.STATUS = ?6)
> OR ((__Z0.STATUS = ?5)
> OR ((__Z0.STATUS = ?4)
> OR ((__Z0.STATUS = ?3)
> OR ((__Z0.STATUS = ?1)
> OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC
> LIMIT 200]
> [05:52:38,036][WARNING][sys-stripe-80-#81][GridContinuousProcessor] Failed
> to wait for ack message. [node=affbdc38-65e0-49f0-9a82-05f1611a95fb,
> routine=35dffec7-068f-4b5c-8e57-2b6f08c0bb04]
> [05:52:38,036][WARNING][long-qry-#221][LongRunningQueryManager] Query
> execution is too long [time=4582ms, type=MAP, distributedJoin=false,
> enforceJoinOrder=true, lazy=false, schema=MediaFan_BladeServices,
> node=TcpDiscoveryNode [id=d9a00801-ec3b-4b38-bbba-7999b5cf428c,
> consistentId=d9a00801-ec3b-4b38-bbba-7999b5cf428c, addrs=ArrayList
> [110.10.123.101], sockAddrs=HashSet [/110.10.123.101:0], discPort=0,
> order=428, intOrder=226, lastExchangeTime=1613988731051, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=256671, segment=0,
> sql='SELECT
> "MediaFan_BladeServices".__Z0._KEY __C0_0,
> "MediaFan_BladeServices".__Z0._VAL __C0_1
> FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
> WHERE (__Z0.TEXTVALUE = ?3) AND ((__Z0.MODULENAME = ?1) AND (__Z0.NAME =
> ?2))', plan=SELECT
> "MediaFan_BladeServices".__Z0._KEY AS __C0_0,
> "MediaFan_BladeServices".__Z0._VAL AS __C0_1
> FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
> /* "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES_NAME_ASC_IDX: NAME =
> ?2 */
> /* scanCount: 95044 */
> WHERE (__Z0.TEXTVALUE = ?3)
> AND ((__Z0.MODULENAME = ?1)
> AND (__Z0.NAME = ?2))]
> [05:52:38,490][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=74e98ce5-3159-4fb6-b5b0-219717460781,
> routine=012d85e2-1e68-429f-bab9-7c33b41ea0ed]
> [05:52:39,162][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=ebc3cde0-72e4-4941-9a2d-855a5ec6c260,
> routine=41ce659a-a56d-4253-820a-ffdd026e4351]
> [05:52:40,292][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=86331a2a-8486-47d7-8cbc-38a08348d7ce,
> routine=48925ed1-34a9-4e9a-8bfd-55759729f712]
> [05:52:40,553][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=d194f864-b8bd-45df-bf04-8d3c2e5f4bac,
> routine=ba749835-ecef-492e-902e-8a606f624ae8]
> [05:52:40,817][WARNING][long-qry-#221][LongRunningQueryManager] Query
> execution is too long [time=6476ms, type=MAP, distributedJoin=false,
> enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
> node=TcpDiscoveryNode [id=92945d9e-387c-467c-a47b-9f3be5b41574,
> consistentId=92945d9e-387c-467c-a47b-9f3be5b41574, addrs=ArrayList
> [110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
> [/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0],
> discPort=0,
> order=361, intOrder=206, lastExchangeTime=1613978781915, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=14040, segment=0,
> sql='SELECT
> "MediaFan_TaskList".__Z0._KEY __C0_0,
> "MediaFan_TaskList".__Z0._VAL __C0_1,
> __Z0.CREATED __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
> ((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
> ((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC LIMIT 200', plan=SELECT
> "MediaFan_TaskList".__Z0._KEY AS __C0_0,
> "MediaFan_TaskList".__Z0._VAL AS __C0_1,
> __Z0.CREATED AS __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> /* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
> /* scanCount: 48739 */
> WHERE (__Z0.STATUS = ?9)
> OR ((__Z0.STATUS = ?8)
> OR ((__Z0.STATUS = ?7)
> OR ((__Z0.STATUS = ?6)
> OR ((__Z0.STATUS = ?5)
> OR ((__Z0.STATUS = ?4)
> OR ((__Z0.STATUS = ?3)
> OR ((__Z0.STATUS = ?1)
> OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC
> LIMIT 200]
> [05:52:40,817][WARNING][long-qry-#221][LongRunningQueryManager] Query
> execution is too long [time=7574ms, type=MAP, distributedJoin=false,
> enforceJoinOrder=true, lazy=false, schema=MediaFan_BladeServices,
> node=TcpDiscoveryNode [id=d9a00801-ec3b-4b38-bbba-7999b5cf428c,
> consistentId=d9a00801-ec3b-4b38-bbba-7999b5cf428c, addrs=ArrayList
> [110.10.123.101], sockAddrs=HashSet [/110.10.123.101:0], discPort=0,
> order=428, intOrder=226, lastExchangeTime=1613988731051, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=256671, segment=0,
> sql='SELECT
> "MediaFan_BladeServices".__Z0._KEY __C0_0,
> "MediaFan_BladeServices".__Z0._VAL __C0_1
> FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
> WHERE (__Z0.TEXTVALUE = ?3) AND ((__Z0.MODULENAME = ?1) AND (__Z0.NAME =
> ?2))', plan=SELECT
> "MediaFan_BladeServices".__Z0._KEY AS __C0_0,
> "MediaFan_BladeServices".__Z0._VAL AS __C0_1
> FROM "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES __Z0
> /* "MediaFan_BladeServices".MEDIAFAN_BLADESERVICES_NAME_ASC_IDX: NAME =
> ?2 */
> /* scanCount: 98066 */
> WHERE (__Z0.TEXTVALUE = ?3)
> AND ((__Z0.MODULENAME = ?1)
> AND (__Z0.NAME = ?2))]
> [05:52:40,817][WARNING][long-qry-#221][LongRunningQueryManager] Query
> execution is too long [time=7853ms, type=MAP, distributedJoin=false,
> enforceJoinOrder=true, lazy=false, schema=MediaFan_TaskList,
> node=TcpDiscoveryNode [id=6e0fba7d-c9a0-4c27-9492-2e231eeb3493,
> consistentId=6e0fba7d-c9a0-4c27-9492-2e231eeb3493, addrs=ArrayList
> [110.10.123.114, 127.0.0.1, 172.17.0.1, 172.20.0.1], sockAddrs=HashSet
> [/110.10.123.114:0, /172.17.0.1:0, /127.0.0.1:0, /172.20.0.1:0],
> discPort=0,
> order=351, intOrder=201, lastExchangeTime=1613877831620, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=true], reqId=39820, segment=0,
> sql='SELECT
> "MediaFan_TaskList".__Z0._KEY __C0_0,
> "MediaFan_TaskList".__Z0._VAL __C0_1,
> __Z0.CREATED __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> WHERE (__Z0.STATUS = ?9) OR ((__Z0.STATUS = ?8) OR ((__Z0.STATUS = ?7) OR
> ((__Z0.STATUS = ?6) OR ((__Z0.STATUS = ?5) OR ((__Z0.STATUS = ?4) OR
> ((__Z0.STATUS = ?3) OR ((__Z0.STATUS = ?1) OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC LIMIT 200', plan=SELECT
> "MediaFan_TaskList".__Z0._KEY AS __C0_0,
> "MediaFan_TaskList".__Z0._VAL AS __C0_1,
> __Z0.CREATED AS __C0_2
> FROM "MediaFan_TaskList".MEDIAFAN_TASKLIST __Z0
> /* "MediaFan_TaskList".MEDIAFAN_TASKLIST.__SCAN_ */
> /* scanCount: 83014 */
> WHERE (__Z0.STATUS = ?9)
> OR ((__Z0.STATUS = ?8)
> OR ((__Z0.STATUS = ?7)
> OR ((__Z0.STATUS = ?6)
> OR ((__Z0.STATUS = ?5)
> OR ((__Z0.STATUS = ?4)
> OR ((__Z0.STATUS = ?3)
> OR ((__Z0.STATUS = ?1)
> OR (__Z0.STATUS = ?2))))))))
> ORDER BY 3 DESC
> LIMIT 200]
> [05:52:41,057][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=619f1567-5a56-4473-8fba-83ee7a82ea07,
> routine=da2db058-b375-4713-8bf6-e788ca315291]
> [05:52:42,281][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=99853e49-dbe6-4641-aa91-809a78c76585,
> routine=4d5d484d-4ec7-4511-9ac8-5e629df6ad09]
> [05:52:51,886][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
> too long JVM pause: 2499 milliseconds.
> [05:52:51,888][WARNING][sys-stripe-31-#32][GridContinuousProcessor] Failed
> to wait for ack message. [node=e09111c4-8826-4a6a-a9b9-6e21c519e055,
> routine=6b66215d-4c72-4be3-8deb-51a91ad8bbe4]
> [05:52:51,890][SEVERE][query-#6221][GridMapQueryExecutor] Failed to execute
> local query.
> class org.apache.ignite.IgniteCheckedException: Failed to execute SQL
> query.
> The database has been closed [90098-197]
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
> at
>
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
> at
>
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed
> [90098-197]
> at
> org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
> at org.h2.message.DbException.get(DbException.java:179)
> at org.h2.message.DbException.get(DbException.java:155)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.engine.Database.checkPowerOff(Database.java:536)
> at org.h2.command.Command.executeQuery(Command.java:228)
> at
>
> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:867)
> ... 13 more
> [05:52:51,899][SEVERE][query-#6186][GridMapQueryExecutor] Failed to execute
> local query.
> class org.apache.ignite.IgniteCheckedException: Failed to execute SQL
> query.
> The database has been closed [90098-197]
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:874)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:955)
> at
>
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:412)
> at
>
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2186)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2139)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3386)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed
> [90098-197]
> at
> org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
> at org.h2.message.DbException.get(DbException.java:179)
> at org.h2.message.DbException.get(DbException.java:155)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.engine.Database.checkPowerOff(Database.java:536)
> at org.h2.command.Command.executeQuery(Command.java:228)
> at
>
> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
> at
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:867)
> ... 13 more
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>