Server Node Stops Unexpectedly

[Alper Tekinalp <al...@evam.com>]

Hi all.

On one of our clusters we have 3 server nodes. 2 times in this week one of
the nodes in cluster stops. We are getting errors that stating the node is
stopping. We listen node joined and left events and before errors we see
logs saying node x left cluster. I suspect that the reason is kinda network
timeout or similar. Because other nodes are up and running.

1 - Is it possible that in a network timeout nodes stops themself?
2 - Is there any parameter or property that we can increase to avoid such
timeouts?
3 - What could be another reason that nodes stops themself?

I will try to collect more information like logs etc.

Regard.

-- 
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>

Re: Server Node Stops Unexpectedly

[dkarachentsev <dk...@gridgain.com>]

Hi,

Node could be segmented if it doesn't respond during some time. The most
common reasons for that:
1) Long GC pause.
2) Operation system could slow down process (f.e. starts swap to disk
process memory).
3) Network issues.

To find out the actual reason you should take GC logs (-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Xloggc:/path/to/gc.log
-XX:+PrintGCApplicationStoppedTime),
dstat (dstat –t --top-mem –m –s –g –d --fs --top-io 3 > dstat.log) and
ignite logs.

Of course you're free to tune timeouts, it worth to start from
IgniteConfiguration.setFailureDetectionTimeout(), by default it is 10 sec.

Thanks!



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Server-Node-Stops-Unexpectedly-tp9189p9198.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.