You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "HEWA WIDANA GAMAGE, SUBASH" <Su...@fmr.com> on 2018/06/11 19:08:43 UTC
Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED
event
Hi,
With default configurations in Ignite 1.9,
Netstat shows about 5000 like following logs. And all sockets closes in about 1 min. And this happens in every 5 mins. This behavior started happening ONLY after a NODE_FAILED event received and we restarted the Ignite(Ignition.stop, Ignition.start) without killing the JVM.
tcp 0 0 1.2.3.4:47500 5.6.7.8:54968 TIME_WAIT
where 1.2.3.4 = node1 IP and 5.6.7.8 = node2 IP.
Just wanted to know whether there's any process known, that can lead to this type of behavior. Can this be because we restart Ignite in the same JVM ?
RE: Ignite opens/close 5000 sockets in every 5mins after
NODE_FAILED event
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
This thread dump is absolutely fine, you confused socket state and java
thread state. These two things are absolutely unrelated.
There should not be so many socket connections (TIME_WAIT means that socket
already closed and waiting for last packages) for three nodes. Could you
please share your configuration and netstat output?
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED
event
Posted by "HEWA WIDANA GAMAGE, SUBASH" <Su...@fmr.com>.
And I took a thread dump by then.
I could see about 10 of following WAITING. Is this looks fine ?
"sys-stripe-6-#29637%null%" #46762 prio=5 os_prio=0 tid=0x00007fa24402b320 nid=0x3900 waiting on condition [0x00007fa22d454000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:526)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:422)
at java.lang.Thread.run(Thread.java:748)
-----Original Message-----
From: HEWA WIDANA GAMAGE, SUBASH [mailto:Subash.HewaWidanaGamage@fmr.com]
Sent: Tuesday, June 12, 2018 2:06 PM
To: user@ignite.apache.org
Subject: RE: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED event
I have only 3 nodes, in three separate physical boxes. Question is, can there be any possibility for creating such number of sockets on 47500, between two nodes ? I cannot reproduce the issue. Hence trying to find any theoretical possibility.
-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com]
Sent: Tuesday, June 12, 2018 11:37 AM
To: user@ignite.apache.org
Subject: RE: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED event
There is no difference on how do you start/stop your node.
Node on start will examine all connections specified in address list: it
takes one address and port and tries to connect to it. If not successfull,
get another address and port. For instance if you have address
1.2.3.4:47500..47509, node will check 10 addresses.
Does this impact you somehow?
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED
event
Posted by "HEWA WIDANA GAMAGE, SUBASH" <Su...@fmr.com>.
I have only 3 nodes, in three separate physical boxes. Question is, can there be any possibility for creating such number of sockets on 47500, between two nodes ? I cannot reproduce the issue. Hence trying to find any theoretical possibility.
-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com]
Sent: Tuesday, June 12, 2018 11:37 AM
To: user@ignite.apache.org
Subject: RE: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED event
There is no difference on how do you start/stop your node.
Node on start will examine all connections specified in address list: it
takes one address and port and tries to connect to it. If not successfull,
get another address and port. For instance if you have address
1.2.3.4:47500..47509, node will check 10 addresses.
Does this impact you somehow?
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite opens/close 5000 sockets in every 5mins after
NODE_FAILED event
Posted by dkarachentsev <dk...@gridgain.com>.
There is no difference on how do you start/stop your node.
Node on start will examine all connections specified in address list: it
takes one address and port and tries to connect to it. If not successfull,
get another address and port. For instance if you have address
1.2.3.4:47500..47509, node will check 10 addresses.
Does this impact you somehow?
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED
event
Posted by "HEWA WIDANA GAMAGE, SUBASH" <Su...@fmr.com>.
We are using TcpDiscoveryVmIpFinder with PARTITIONED cache with one backup.
But I specifically noticed this behavior ONLY after received a NODE_FAILED event on the node, and then we restart Ignite Process in the same JVM. So is there any known/possible implications of restarting Ignite without killing the JVM ?
-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com]
Sent: Tuesday, June 12, 2018 4:58 AM
To: user@ignite.apache.org
Subject: Re: Ignite opens/close 5000 sockets in every 5mins after NODE_FAILED event
Hi,
TcpDiscoveryMulticastIpFinder produces such a big number of connections. I'd
recommend to switch to TcpDiscoveryVmIpFinder with static set of addresses.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite opens/close 5000 sockets in every 5mins after
NODE_FAILED event
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
TcpDiscoveryMulticastIpFinder produces such a big number of connections. I'd
recommend to switch to TcpDiscoveryVmIpFinder with static set of addresses.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/