You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by BEELA GAYATRI <be...@tcs.com> on 2020/12/02 10:46:22 UTC
RE: Getting error Node is out of topology (probably, due to
short-time network problems)
Hi Team,
As suggested , we have incorporated below changes for running the nodes. Still we are getting “Getting error Node is out of topology (probably, due to short-time network problems)”. Also the data in the cache is lost every time the cache node being stopped
-Xms 512M
-Xmx5G
-XX:+AlwaysPreTouch
-XX:+UseG1GC
-XX:+ScavengeBeforeFullGC
-XX:+DisableExplicitGC
-DIGNITE_JVM_PAUSE_DETECTOR_THRESHOLD=5000
We are having 16 nodes as data grid nodes/Computation nodes with each server having 4 CPU’s with 5GB RAM.
Attaching configuration file and log files and GC log files.
We are observing this behavior when nodes are idle(no cache operation/no computation ) for some time (from few hours to few days). Please suggest
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: BEELA GAYATRI<ma...@tcs.com>
Sent: Thursday, November 26, 2020 10:09 AM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: RE: Getting error Node is out of topology (probably, due to short-time network problems)
"External email. Open with Caution"
Hi Igor,
Asper the belowsuggesion, we have incorporated jvm property as below and run all the 16 nodes.
“-DIGNITE_JVM_PAUSE_DETECTOR_THRESHOLD=10000”
Even though one of the node is out of topology and cache was stopped . PFA GClog and Ignite log for the same. Please suggest what can be done further.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: ibelyakov<ma...@gmail.com>
Sent: Monday, November 23, 2020 3:03 PM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: RE: Getting error Node is out of topology (probably, due to short-time network problems)
"External email. Open with Caution"
Hi,
According to the provided log I see "Blocked system-critical thread has been
detected" message and that the node was segmented since it was unable to
respond to another node. Most probably it's caused by JVM pauses, possibly
related with GC.
Do you collect GC logs for the nodes?
You can find an information how to enable GC logs here:
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/troubleshooting#detailed-gc-logs
Igor
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
RE: Getting error Node is out of topology (probably, due to
short-time network problems)
Posted by BEELA GAYATRI <be...@tcs.com>.
Hi Igor,
We have Observed the CPU and memory utilizations on the servers where these nodes are running and the CPU utilizations are very less . Still we are observing the issue (sometimes we are getting jvm pauses after that node is getting out of topology, sometimes without jvm pauses node is getting out of topology)
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: ibelyakov<ma...@gmail.com>
Sent: Tuesday, December 15, 2020 9:19 PM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: RE: Getting error Node is out of topology (probably, due to short-time network problems)
"External email. Open with Caution"
Hi,
According to the provided GC logs I don't see anything suspicious.
Do you run Ignite nodes on VMs? If yes, do you have monitoring and is it
possible to check CPU usage during period of time when the issue happend?
Regards,
Igor
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
RE: Getting error Node is out of topology (probably, due to
short-time network problems)
Posted by ibelyakov <ig...@gmail.com>.
Hi,
According to the provided GC logs I don't see anything suspicious.
Do you run Ignite nodes on VMs? If yes, do you have monitoring and is it
possible to check CPU usage during period of time when the issue happend?
Regards,
Igor
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/