You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by zshamrock <al...@gmail.com> on 2017/10/09 06:31:07 UTC
Local node was dropped from cluster due to network problems
cascading failures
Hi, we are running cluster of 2 Ignite 1.9 servers running on the EC2. The
EC2 instances are r4.large, i.e. 16GB of memory each. We use Amazon S3 based
discovery for both servers and clients.
We have another EC2 instance (r4.large, 16GB) where our app service is
running, and where the Ignite clients live. There are 5 Ignite clients are
running there (as we run the app in the docker container using
`network_mode: host`), and so there are 5 docker instances with the app are
running.
We also set the socket timeout for the `TcpDiscoverySpi` as the recommended
for the EC2 to 30 seconds for both servers and clients.
The problem is that after some period of time we get `Local node failed`
error, and the looks like the cluster becomes unstable, as it reports in a
loop the new topology version increased constantly, i.e. cascading failure.
```
2017-10-08 17:04:14.520 WARN 6 --- [tcp-client-disco-msg-worker-#4%st%]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Local node was dropped from
cluster due to network problems, will try to reconnect with new id after
10000ms (reconnect delay can be changed using
IGNITE_DISCO_FAILED_CLIENT_RECONNECT_DELAY system property)
[newId=85e37c0f-fd44-430f-9247-06f783589523,
prevId=48e71e9f-7548-460b-9320-2155be8a30a4, locNode=TcpDiscoveryNode
[id=48e71e9f-7548-460b-9320-2155be8a30a4, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 172.17.0.1, 172.31.29.171],
sockAddrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:0], discPort=0,
order=138, intOrder=0, lastExchangeTime=1507193821071, loc=true,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=true],
nodeInitiatedFail=e5897e87-65e8-4bf8-947e-7b3f244c3458,
msg=TcpCommunicationSpi failed to establish connection to node
[rmtNode=TcpDiscoveryNode [id=48e71e9f-7548-460b-9320-2155be8a30a4,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 172.31.29.171],
sockAddrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:0], discPort=0,
order=138, intOrder=74, lastExchangeTime=1507392564555, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=true], errs=class
o.a.i.IgniteCheckedException: Failed to connect to node (is node still
alive?). Make sure that each ComputeTask and cache Transaction has a timeout
set in order to prevent parties from waiting forever in case of network
issues [nodeId=48e71e9f-7548-460b-9320-2155be8a30a4,
addrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:47103,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:47103,
/0:0:0:0:0:0:0:1%lo:47103, /127.0.0.1:47103]], connectErrs=[class
o.a.i.IgniteCheckedException: Failed to connect to address:
ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:47103, class
o.a.i.IgniteCheckedException: Failed to connect to address:
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:47103, class
o.a.i.IgniteCheckedException: Failed to connect to address:
/0:0:0:0:0:0:0:1%lo:47103, class o.a.i.IgniteCheckedException: Failed to
connect to address: /127.0.0.1:47103]]]
2017-10-08 17:04:24.888 WARN 6 --- [tcp-client-disco-msg-worker-#4%st%]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Client node was reconnected after
it was already considered failed by the server topology (this could happen
after all servers restarted or due to a long network outage between the
client and servers). All continuous queries and remote event listeners
created by this client will be unsubscribed, consider listening to
EVT_CLIENT_NODE_RECONNECTED event to restore them.
2017-10-08 17:04:24.981 INFO 6 --- [disco-event-worker-#23%st%]
o.a.i.i.m.d.GridDiscoveryManager : Client node reconnected to
topology: TcpDiscoveryNode [id=85e37c0f-fd44-430f-9247-06f783589523,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 172.31.29.171],
sockAddrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:0], discPort=0,
order=188, intOrder=0, lastExchangeTime=1507193821071, loc=true,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=true]
2017-10-08 17:04:24.988 INFO 6 --- [disco-event-worker-#23%st%]
o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=188,
servers=2, clients=8, CPUs=12, heap=17.0GB]
2017-10-08 17:04:47.264 WARN 6 --- [tcp-client-disco-msg-worker-#4%st%]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Received EVT_NODE_FAILED event
with warning [nodeInitiatedEvt=TcpDiscoveryNode
[id=28db9f51-f3a3-42d2-b241-520de1124d77, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 172.17.0.1, 172.31.22.48],
sockAddrs=[ip-172-31-22-48.us-west-2.compute.internal/172.31.22.48:47500,
ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:47500,
/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1507482264715, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], msg=TcpCommunicationSpi
failed to establish connection to node [rmtNode=TcpDiscoveryNode
[id=691db97e-1bb0-49d9-aa8c-a5c6114e4842, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 172.17.0.1, 172.31.29.171],
sockAddrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:0], discPort=0,
order=186, intOrder=98, lastExchangeTime=1507466513487, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=true], errs=class
o.a.i.IgniteCheckedException: Failed to connect to node (is node still
alive?). Make sure that each ComputeTask and cache Transaction has a timeout
set in order to prevent parties from waiting forever in case of network
issues [nodeId=691db97e-1bb0-49d9-aa8c-a5c6114e4842,
addrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:47104,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:47104,
/0:0:0:0:0:0:0:1%lo:47104, /127.0.0.1:47104]], connectErrs=[class
o.a.i.IgniteCheckedException: Failed to connect to address:
ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:47104, class
o.a.i.IgniteCheckedException: Failed to connect to address:
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:47104, class
o.a.i.IgniteCheckedException: Failed to connect to address:
/0:0:0:0:0:0:0:1%lo:47104, class o.a.i.IgniteCheckedException: Failed to
connect to address: /127.0.0.1:47104]]]
2017-10-08 17:04:47.274 WARN 6 --- [disco-event-worker-#23%st%]
o.a.i.i.m.d.GridDiscoveryManager : Node FAILED: TcpDiscoveryNode
[id=691db97e-1bb0-49d9-aa8c-a5c6114e4842, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 172.17.0.1, 172.31.29.171],
sockAddrs=[ip-172-17-0-1.us-west-2.compute.internal/172.17.0.1:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
ip-172-31-29-171.us-west-2.compute.internal/172.31.29.171:0], discPort=0,
order=186, intOrder=98, lastExchangeTime=1507482264827, loc=false,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=true]
2017-10-08 17:04:47.278 INFO 6 --- [disco-event-worker-#23%st%]
o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=189,
servers=2, clients=7, CPUs=12, heap=17.0GB]
...
```
What could be the cause of this "Local node was dropped from cluster due to
network problems" (and why after it happened the cluster seems unstable) and
what are the strategies to resolve it?
One thing we plan is to create another EC2 instance and split the Ignite
clients now between 2 EC2 instances, but it is good to know the root cause
of the problem anyway, as not necessary this split will help.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Local node was dropped from cluster due to network problems
cascading failures
Posted by zshamrock <al...@gmail.com>.
Ok, we identified the root cause. It was not specifically related to the
Ignite, but rather the security settings (EC2 security group), i.e. we only
had inbound port 47100 open to the EC2 instance. But as you can see from the
original message, the error is about the nodes running on ports 47103 and
47104, actually all others except 47100.
There is `TcpCommunicationSpi`
https://apacheignite.readme.io/v1.9/docs/network-config#section-configuration,
which defines `setLocalPort` (default to 47100) and `setLocalPortRange`
which is 100. And my assumption is that because we are running multiple
services on the same machine, so every Ignite client will get its own port
starting from 47100, and up to 47200 (or 47199?) (see `setLocalPortRange`
above). So as we are running multiple of the them, only one will get 47100
port, others will get 47101, and 47102 (as we have max of 3 running on the
same machine currently), and so on.
And they connect to the server node, which is listening port 47500 (which is
opened in the security group to connect).
So during the cluster start up everything works fine.
But then because ports 47101-... were not open on our app side, the server
could not reach back other clients apart from the one running on port 47100.
This is my theory (but at least opening those ports fixed the problem).
Of course, there is still an open question, is why the client node starts to
fail only when there is a load, I would expect there is a periodic heart
beat, so the server should not reach the client node almost immediately
after the cluster started (I mean the client nodes listening on ports
14701-...).
But we only start see the error after couple of hours when the system is in
use.
Could you, please, comment on this?
Thank you.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Local node was dropped from cluster due to network problems
cascading failures
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Please find how to gather GC logs here [1].
[1]
https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats
On Wed, Oct 11, 2017 at 5:00 PM, zshamrock <al...@gmail.com>
wrote:
> Hi, Andrew. Thank you for the reply.
>
> We will definitely try this.
>
> Another thing we today switched to the static IP list for the TCP IP finder
> instead of S3 based discovery, to see whether that could be the problem as
> well. So, will test static IP based discovery for today. If it didn't make
> any difference, then will try the option you've suggested.
>
> Will keep you updated.
>
> Also could you, please, point into the direction/source of how to check GC
> logs for the Ignite client/server?
>
> Thank you once again.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Local node was dropped from cluster due to network problems
cascading failures
Posted by zshamrock <al...@gmail.com>.
Hi, Andrew. Thank you for the reply.
We will definitely try this.
Another thing we today switched to the static IP list for the TCP IP finder
instead of S3 based discovery, to see whether that could be the problem as
well. So, will test static IP based discovery for today. If it didn't make
any difference, then will try the option you've suggested.
Will keep you updated.
Also could you, please, point into the direction/source of how to check GC
logs for the Ignite client/server?
Thank you once again.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Local node was dropped from cluster due to network problems
cascading failures
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
There is a known bug mixing IPv6 and IPv4 protocol can cause node failures.
Try to set -Djava.net.preferIpv4Stack=true JVM option for all nodes.
Also it is possible clients do smth that cause high GC pressure on server
node.
Would you please check GC logs?
On Wed, Oct 11, 2017 at 12:14 PM, zshamrock <al...@gmail.com>
wrote:
> Split the services into 2 separate instances, one with high intensive
> network
> usage, and the one with where only Ignite clients are running didn't help,
> the error still persists and happens after couple hours of working.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: Local node was dropped from cluster due to network problems
cascading failures
Posted by zshamrock <al...@gmail.com>.
Split the services into 2 separate instances, one with high intensive network
usage, and the one with where only Ignite clients are running didn't help,
the error still persists and happens after couple hours of working.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Local node was dropped from cluster due to network problems
cascading failures
Posted by zshamrock <al...@gmail.com>.
Could it be that the network bandwidth is reached on the EC2 instance?
r4.large has up to 10Gigabit network performance, and we are running another
network extensive service on those machines (although split into the
separate instance today), and according to the Cloudwatch we are about the
limit of the network capacity, which probably could affect the Ignite
clients talking to the server.
Could it be?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/