Partition eviction failed (current node is stopping)

[Naveen <na...@gmail.com>]

HI

I am usng 2.9.0, node is joining the cluster and immediately getting removed
from the cluster
When do we get these errors and how do we resolve them

2020-12-17 16:54:18,948][ERROR][tcp-disco-msg-worker-[70cbe8b9
10.119.10.64:47500]-#2][G] Blocked system-critical thread has been detected.
This can lead to cluster-wide undefined behaviour
[workerName=grid-nio-worker-tcp-comm-1,
threadName=grid-nio-worker-tcp-comm-1-#25, blockedFor=20s]
[2020-12-17 16:54:18,948][WARN ][tcp-disco-msg-worker-[70cbe8b9
10.119.10.64:47500]-#2][G] Thread [name="grid-nio-worker-tcp-comm-1-#25",
id=58, state=RUNNABLE, blockCnt=0, waitCnt=0]

[2020-12-17 16:54:18,949][WARN ][tcp-disco-msg-worker-[70cbe8b9
10.119.10.64:47500]-#2][root] Possible failure suppressed accordingly to a
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-1,
igniteInstanceName=null, finished=false, heartbeatTs=1608209658947]]]

Seeing these messages for all the caches. why does it evict when we have
still enough RAM to load, 

[2020-12-17 14:11:29,537][INFO ][sys-#62][PartitionsEvictManager] Eviction
in progress [permits=1, threads=2, groups=6, remainingPartsToEvict=331]

Any pointers would be great help

Thanks
Naveen




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Partition eviction failed (current node is stopping)

[akurbanov <an...@gmail.com>]

Hello Naveen,

Could you please post the full log?

I suspect that the node where you see this messages was offline for some
time and now full rebalance is currently in progress, which evicts
partitions at the first stage (based on the "remainingPartsToEvict=331"
part).

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/