You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by VeenaMithare <v....@cmcmarkets.com> on 2020/07/30 10:19:52 UTC
2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side
Hello,
When do EVT_NODE_SEGMENTED event get generated on the client side in 2.8.1 ?
I noticed that in 2.7.6 , when all the servers are restarted and the client
reconnects again with a new node id - it used to receive a
EVT_NODE_SEGMENTED . In 2.8.1 it receives EVT_NODE_RECONNECTED.
Kindly confirm my understanding.
regards,
Veena.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Hi Anton,
Thank you for the reply .
>>I can confirm that there is no SEGMENTED event thrown on 2.8.1.
I guess you meant that there is no SEGMENTED event thrown in 2.8.1 client
when the servers are brought down and started again. Because SEGMENTED event
could be thrown in 2.8.1 in other scenarios as well I guess, like a long GC
pause .
I have captured the logs for 2.7.6 client , where it receives a SEGMENTED
event. On closer look, it looks like the client's attempt to reconnect
failed in this case with 'Authentication Failure' and this has caused the
SEGMENTED event.
Check this part of the log in the attached log :
2020-10-08 09:59:33,327
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
ERROR
com.company.projectname.project.common.plugin.discoveryspi.CustomTcpDiscoverySpi
[] - Failed to reconnect, segment local node.
org.apache.ignite.spi.IgniteSpiException: Authentication failed
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
2020-10-08 09:59:33,328
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
DEBUG
In 2.8.1, we have changed the security plugin , hence looks like the client
is able to reconnect successfully.
In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of node needed, since the node is already connected )
2.txt <http://apache-ignite-users.70518.x6.nabble.com/file/t2757/2.txt>
regards,
Veena
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Thanks Ilya,
>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.
In that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.
Does the server or the client need to have a handler for any other 'cluster
specific' events ?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You can just set clientReconnectEnabled to false. Then, if it is out of
topology, just start a new one to make sure you have a fresh start.
Not sure about other events. Maybe handle ACTIVATED/DEACTIVATED?
Regards,
--
Ilya Kasnacheev
вт, 13 окт. 2020 г. в 14:49, VeenaMithare <v....@cmcmarkets.com>:
> Thanks Ilya,
>
> >>I don't think so. Node can reconnect to a *different* cluster, or it may
> be disconnected for a while and return to the same cluster. For client
> node,
> segmented/disconnected difference is not relevant. Neither it is for server
> node, in fact.
>
> In that case, we will have the same handler for segmented and reconnected
> events. The handler will restart ignite and deploy any continuous queries
> as
> needed.
> Thank you for this guidance.
>
> Does the server or the client need to have a handler for any other 'cluster
> specific' events ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Thanks Ilya,
>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.
In that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.
Does the server or the client need to have a handler for any other 'cluster
specific' events ?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I don't think so. Node can reconnect to a *different* cluster, or it may be
disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.
Regards,
--
Ilya Kasnacheev
вт, 13 окт. 2020 г. в 10:18, VeenaMithare <v....@cmcmarkets.com>:
> Hi Team,
>
>
> In terms of the client, do we need to have the below 2 EVENT handlers? :
> 1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
> queries if needed.
> 2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
> restart of ignite needed, since the node is already connected )
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Hi Team,
In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of ignite needed, since the node is already connected )
regards,
Veena.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Hi Anton,
Thank you for the reply .
>>I can confirm that there is no SEGMENTED event thrown on 2.8.1.
I guess you meant that there is no SEGMENTED event thrown in 2.8.1 client
when the servers are brought down and started again. Because SEGMENTED event
could be thrown in 2.8.1 in other scenarios as well I guess, like a long GC
pause .
I have captured the logs for 2.7.6 client , where it receives a SEGMENTED
event. On closer look, it looks like the client's attempt to reconnect
failed in this case with 'Authentication Failure' and this has caused the
SEGMENTED event.
Check this part of the log in the attached log :
2020-10-08 09:59:33,327
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
ERROR
com.company.projectname.project.common.plugin.discoveryspi.CustomTcpDiscoverySpi
[] - Failed to reconnect, segment local node.
org.apache.ignite.spi.IgniteSpiException: Authentication failed
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
2020-10-08 09:59:33,328
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
DEBUG
In 2.8.1, we have changed the security plugin , hence looks like the client
is able to reconnect successfully.
In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of node needed, since the node is already connected )
SegmentedClientLogs.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t2757/SegmentedClientLogs.txt>
regards,
Veena
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by akurbanov <an...@gmail.com>.
Hi Veena,
I can confirm that there is no SEGMENTED event thrown on 2.8.1.
I cannot reproduce catching the SEGMENTED event on 2.7.6. Could you help to
reproduce the behaviour or share the log where this event might be noticed?
Best regards,
Anton
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Hi ,
Kindly awaiting a reply on this ,
regards,
Veena
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
I rechecked this. My cluster setup is :
servers : 3
client : 1
Bring down all nodes in the cluster for about 2 mins (
failuredetectiontimeout is 10 seconds ). The client receives
EVT_NODE_RECONNECTED and does not receive EVT_NODE_SEGMENTED.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Hi Team,
Kindly awaiting a reply on this,
regards,
Veena.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
Hi Anton,
There are a few thoughts on this :
1. I found this issue after I upgraded from 2.7.6 to 2.8.1 ( No timeout
values etc has been changed in our application ). So for the same scenario
and the same application configuration
a. in 2.7.6 , the client receives a evt_node_segmented
b. in 2.8.1 , the client receives a evt_node_reconnected.
2. >>>>it logs the event of client state updated
from DISCONNECTED to RECONNECTED because the node succeeded to join the
topology back within some time, the node was not segmented from the
topology
From the logs I posted, it looks like client tries to reconnect using a new
client id
>>[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Client node
disconnected from cluster, will try to reconnect with new id
3. The logs also say
>>Client node was
reconnected after it was already considered failed by the server topology
(this could happen after all servers restarted or due to a long network
outage between the client and servers). All continuous queries and remote
event listeners created by this client will be unsubscribed, consider
listening to EVT_CLIENT_NODE_RECONNECTED event to restore them.
If a client node reconnects after it was considered failed by server,
shouldnt it receive a EVT_NODE_SEGMENTED ?
regards,
Veena.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by akurbanov <an...@gmail.com>.
Hello,
This case seems to be correct, it logs the event of client state updated
from DISCONNECTED to RECONNECTED because the node succeeded to join the
topology back within some time, the node was not segmented from the
topology. What timeouts do you use in your nodes configuration?
I would suggest to perform a test with different timeouts and check that if
node stays offline for too long, the node gets segmented from the topology.
Best regards,
Anton
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by VeenaMithare <v....@cmcmarkets.com>.
HI Anton,
Checked again -In 2.8.1 it receives the EVT_NODE_RECONNECTED with a new node
id . It receives Eventtype 17 ( EVT_NODE_RECONNECTED ) and not Eventtype
14 ( EVT_NODE_SEGMENTED ) .
Also it connects with a new node id. Please find the log with relevant parts
highlighted.
2020-07-31 06:32:17,150
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - */Client node
disconnected from cluster, will try to reconnect with new id/*
[newId=bcbaa5ad-07aa-4118-802e-433dcfcb83b4,
prevId=bca9b2da-bca5-4eac-817b-c989a072b50a, locNode=TcpDiscoveryNode
[id=bca9b2da-bca5-4eac-817b-c989a072b50a,
consistentId=bca9b2da-bca5-4eac-817b-c989a072b50a, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[machinename/a.b.c.d:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0,
order=4, intOrder=0, lastExchangeTime=1596173504995, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true]]
2020-07-31 06:32:46,197
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Router node:
TcpDiscoveryNode [id=4ca9ed4f-416b-43f2-ada9-d5569e7fe8ce,
consistentId=d46c9ddd-d97c-4d07-b857-fc4281d9f495, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[/0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501, machinename/a.b.c.d:47501],
discPort=47501, order=3, intOrder=3, lastExchangeTime=1596173566191,
loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false]
2020-07-31 06:32:46,232
[disco-notifier-worker-#96%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl [] -
Assigned mvcc coordinator [crd=MvccCoordinator
[topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0],
nodeId=d0b9191e-18ea-42fe-8148-b7d28a75d9f8, ver=1596173562700, local=false,
initialized=false]]
2020-07-31 06:32:46,233
[exchange-worker-#136%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.processors.cache.GridCacheProcessor [] - Stopped
cache [cacheName=ignite-sys-cache]
2020-07-31 06:32:46,233
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] WARN
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - /*Client node was
reconnected after it was already considered failed by the server topology
(this could happen after all servers restarted or due to a long network
outage between the client and servers). All continuous queries and remote
event listeners created by this client will be unsubscribed, consider
listening to EVT_CLIENT_NODE_RECONNECTED event to restore them.*/
[06:32:46] Client node was reconnected after it was already considered
failed by the server topology (this could happen after all servers restarted
or due to a long network outage between the client and servers). All
continuous queries and remote event listeners created by this client will be
unsubscribed, consider listening to EVT_CLIENT_NODE_RECONNECTED event to
restore them.
2020-07-31 06:32:46,237
[disco-event-worker-#98%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
Client node reconnected to topology: TcpDiscoveryNode
[id=bcbaa5ad-07aa-4118-802e-433dcfcb83b4,
consistentId=bca9b2da-bca5-4eac-817b-c989a072b50a, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[machinename/a.b.c.d:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0,
order=4, intOrder=0, lastExchangeTime=1596173504995, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true]
[06:32:46] Topology snapshot [ver=4, locNode=bcbaa5ad, servers=3, clients=1,
state=INACTIVE, CPUs=32, offheap=12.0GB, heap=28.0GB]
[06:32:46] ^-- Baseline [id=0, size=3, online=3, offline=0]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
Topology snapshot [ver=4, locNode=bcbaa5ad, servers=3, clients=1,
state=INACTIVE, CPUs=32, offheap=12.0GB, heap=28.0GB]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] INFO
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
^-- Baseline [id=0, size=3, online=3, offline=0]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] DEBUG
SegmentationEventListener [] - /*Received Event 17 Event Node Id
bcbaa5ad-07aa-4118-802e-433dcfcb83b4 for Node :
bcbaa5ad-07aa-4118-802e-433dcfcb83b4*/
regards,
Veena.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client
side
Posted by akurbanov <an...@gmail.com>.
Hello Veena,
Could you please double-check if the situation is the same? Reconnected
event should be fired if node reconnects with the same id, let me go through
the changes that could possibly affect that.
Best regards,
Anton
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/