You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2022/12/19 17:47:06 UTC
[GitHub] [ignite] kinjal63 opened a new issue, #10452: Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT
kinjal63 opened a new issue, #10452:
URL: https://github.com/apache/ignite/issues/10452
I run Ignite as a local node within my java application process where I've observed a few Ignite crashes as mentioned below intermittently. In fact, I'm using Ignite as a SQL engine and around 100k records are persisted on an average and a continuous stream of data is getting updated for a given primary key in a record. Ignite is running in a persistent mode with a low memory footprint.
Error Example 1:
ERROR [tcp-disco-msg-worker] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, heartbeatTs=1671441824592]]]
Error Example 2:
ERROR [node-stopper] Stopping local node on Ignite failure: [failureCtx=FailureContext [type=SYSTEM_CRITICAL_OPERATION_TIMEOUT, err=class o.a.i.IgniteException: Checkpoint read lock acquisition has been timed out.]]
What could be the potential issue here? I've tried put in required config such as systemWorkerBlockedTimeout set to ~30sec and enabled page throtting for checkpointing process but that doesn't help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] doerejmi commented on issue #10452: Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT
Posted by "doerejmi (via GitHub)" <gi...@apache.org>.
doerejmi commented on issue #10452:
URL: https://github.com/apache/ignite/issues/10452#issuecomment-1558824095
I'm also seeing the same error message and losing the node. It happens under fairly high load (many new cache objects added to a cache from multiple threads). The issue does not occur when starting only a single node. The thread reporting the block changes but mostly the IgniteTxManager in the stacktrace.
It happens in both 2.12 and 2.15 release versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT [ignite]
Posted by "sk0x50 (via GitHub)" <gi...@apache.org>.
sk0x50 commented on issue #10452:
URL: https://github.com/apache/ignite/issues/10452#issuecomment-1779368806
Closed due to inactivity. Unfortunately, the issue cannot be analyzed without a thread dump at least. The answer that is provided by @virendar5Kmr can be used as a workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] virendar5Kmr commented on issue #10452: Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT
Posted by "virendar5Kmr (via GitHub)" <gi...@apache.org>.
virendar5Kmr commented on issue #10452:
URL: https://github.com/apache/ignite/issues/10452#issuecomment-1434655646
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="checkpointFrequency" value="300000"/>
increase the checkpoint frequency and try if this help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT [ignite]
Posted by "sk0x50 (via GitHub)" <gi...@apache.org>.
sk0x50 closed issue #10452: Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT
URL: https://github.com/apache/ignite/issues/10452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] shishkovilja commented on issue #10452: Ignite errors out on SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT
Posted by "shishkovilja (via GitHub)" <gi...@apache.org>.
shishkovilja commented on issue #10452:
URL: https://github.com/apache/ignite/issues/10452#issuecomment-1514958852
Hi @kinjal63 ,
Which version of Ignite is used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org