You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by zork <pr...@gmail.com> on 2020/06/12 06:29:28 UTC
Questions Regarding Critical Workkers Health Check
Hello Ignite Experts,
We have recently upgraded from version 2.6 to 2.8.0 and have started to face
some weird behavior since then.
With the below configuration:
<property name="metricsUpdateFrequency" value="1000"/>
<property name="failureDetectionTimeout" value="1100"/>
<property name="clientFailureDetectionTimeout" value="1000"/>
We are seeing the below log (with different thread names) multiple times
every second as soon as the ignite server is started:
06-11 08:40:50,309978 [61] ERROR G(Ignite) - Blocked system-critical thread
has been detected. This can lead to cluster-wide undefined behaviour
[workerName=grid-nio-worker-tcp-comm-3,
threadName=grid-nio-worker-tcp-comm-3-#27%IgniteCluster1%, blockedFor=1s]
06-11 08:40:50,310267 [61] WARN G(Ignite) - Thread
[name="grid-nio-worker-tcp-comm-3-#27%IgniteCluster1%", id=40,
state=RUNNABLE, blockCnt=0, waitCnt=0]
06-11 08:40:50,310609 [61] WARN (Ignite) - Possible failure suppressed
accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=grid-nio-worker-tcp-comm-3, igniteInstanceName=IgniteCluster1,
finished=false, heartbeatTs=1591864848869]]]
06-11 08:40:50,311975 [61] WARN CacheDiagnosticManager(Ignite) - Page locks
dump:
Thread=[name=NgServiceProvider_EventsExecutor_0, id=77], state=WAITING
Locked pages = []
Locked pages log: name=NgServiceProvider_EventsExecutor_0
time=(1591864850311, 2020-06-11 14:10:50.311)
Thread=[name=exchange-worker-#43%IgniteCluster1%, id=63],
state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#43%IgniteCluster1%
time=(1591864850311, 2020-06-11 14:10:50.311)
Thread=[name=sys-#45%IgniteCluster1%, id=65], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#45%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)
Thread=[name=sys-#48%IgniteCluster1%, id=68], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#48%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)
Thread=[name=sys-#49%IgniteCluster1%, id=69], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#49%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)
Thread=[name=sys-#51%IgniteCluster1%, id=71], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=sys-#51%IgniteCluster1% time=(1591864850311,
2020-06-11 14:10:50.311)
Full Logs attached here:
WebIgniteService_WEB.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2754/WebIgniteService_WEB.log>
However, if I change my timeouts like this:
<property name="metricsUpdateFrequency" value="2000"/>
<property name="failureDetectionTimeout" value="10000"/>
<property name="clientFailureDetectionTimeout" value="30000"/>
It still occurs but a lot less frequently (I observed it only after I have
added 1 server and 5 clients and the communication started between them).
I did some research and found that this is related to the Critical Workers
Health Check feature which I think is a great addition to ignite but I have
a few questions regarding the same.
Question-1: Does the above log mean that the thread is really blocked? or is
it just busy doing something else?
Question-2: How can I decide the suitable values of these timeouts for my
case? The former values were working for me earlier but now I face this
exception.
Question-3: I can see that this failure type (WORKER_THREAD_BLOCKED) is
actually ignored by default so why do we still see it as an ERROR in logs?
Question-4: As a remedy to this, I have thought of adding another timeout to
my configuration:
<property name="systemWorkerBlockedTimeout" value="30000"/>
I read that failureDetectionTimeout is ignored in case any other timeout is
set. Would that mean now my failure detection timeout would also become
30000? or would it mean that failureDetectionTimeout would still be the
configured value and just that it's value will be ignored for
systemWorkerBlockedTimeout (which would now be 30000)?
Question-5: How to decide the value for systemWorkerBlockedTimeout, do we
have some guidelines here?
Question-6: As I can see in
https://issues.apache.org/jira/browse/IGNITE-10154, this
WORKER_THREAD_BLOCKED failure is ignored by default, but on setting some
positive value for systemWorkerBlockedTimeout, it would actually start
working. However, I'm not sure if I want that right now. How else can I
handle this scenario so that I don't get these unnecessary and very frequent
exceptions without enabling this failure?
Please correct me if I'm wrong anywhere.
Thanks in advance.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Questions Regarding Critical Workkers Health Check
Posted by akorensh <al...@gmail.com>.
Hi,
Question-1: Does the above log mean that the thread is really blocked? or
is
it just busy doing something else?
The thread in question might be busy doing something else and did not
update its heartbeat timestamp.
See:
https://apacheignite.readme.io/docs/critical-failures-handling#critical-workers-health-check
When that happens a thread dump is generated and the pre-configured
failurehandler is called:
Question-2: How can I decide the suitable values of these timeouts for my
case? The former values were working for me earlier but now I face this
exception.
Usually the default values are what works best, otherwise use
experimental means to determine optimal settings for your use case.
Question-3: I can see that this failure type (WORKER_THREAD_BLOCKED) is
actually ignored by default so why do we still see it as an ERROR in logs?
When a thread is blocked, the event in question is: WORKER_THREAD_BLOCKED
The error message reflects the event type. It is up to the failure handler
to handle that
event. See:
https://apacheignite.readme.io/docs/critical-failures-handling#failure-handling(see
implementation below)
Question-4: As a remedy to this, I have thought of adding another timeout
to
my configuration:
<property name="systemWorkerBlockedTimeout" value="30000"/>
I read that failureDetectionTimeout is ignored in case any other timeout
is
set. Would that mean now my failure detection timeout would also become
30000? or would it mean that failureDetectionTimeout would still be the
configured value and just that it's value will be ignored for
systemWorkerBlockedTimeout (which would now be 30000)?
Setting systemWorkerBlockedTimeout affects the timeout related to that
property, and nothing else.
Question-5: How to decide the value for systemWorkerBlockedTimeout, do we
have some guidelines here?
Leave default or use experimental means.
Question-6: As I can see in
https://issues.apache.org/jira/browse/IGNITE-10154, this
WORKER_THREAD_BLOCKED failure is ignored by default, but on setting some
positive value for systemWorkerBlockedTimeout, it would actually start
working. However, I'm not sure if I want that right now. How else can I
handle this scenario so that I don't get these unnecessary and very frequent
exceptions without enabling this failure?
This was an issue w/an old verison (2.7) and is now resolved.
Setting systemWorkerBlockedTimeout would only affect that property only
Liveness check is enabled either way and failurehandling is the sole domain
of the configured
Failure Handler.
Here are some links to the implementation to make it clearer:
All relevant threads get put into a workers registry:
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/worker/WorkersRegistry.java
the registry get started by the kernel:
https://github.com/apache/ignite/blob/c3a2deb8f464e4547f65164d2ad62b10854cb199/modules/core/src/main/java/org/apache/ignite/internal/IgnitionEx.java#L1805
the lamda uses the FailureProcessor to handle failures using the configured
FailureHandler (default StopNodeOrHaltFaiulreHandler):
https://github.com/apache/ignite/blob/c3a2deb8f464e4547f65164d2ad62b10854cb199/modules/core/src/main/java/org/apache/ignite/internal/processors/failure/FailureProcessor.java#L156
The workers registry continuously monitors all threads here:
Take a look at the error message and how the thread dump is generated.
https://github.com/apache/ignite/blob/c3a2deb8f464e4547f65164d2ad62b10854cb199/modules/core/src/main/java/org/apache/ignite/internal/worker/WorkersRegistry.java#L175
Thanks, Alex
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/