You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Jason <fq...@outlook.com> on 2016/08/11 14:48:34 UTC

Fail to join topology and repeat join process

hi Ignite team,

In a cluster with 20 server nodes, I manually restarted a server node to
test its partition re-balance and reliability, then the restarted node
couldn't join the topology with the below error. And this process lasted for
few hours, but still couldn't move forward.

The attached is the log in the other remote server nodes.

FYI, we've a big cache with 20G off_heap memory per node.

Would you like to take a look and give us some suggestion on how to tune
this? 

Any suggestion or advice will be appreciated.

Thanks,
-Jason

[TcpDiscoverySpi] Node has not been connected to topology and will repeat
join process. Check remote nodes logs for possible error messages. Note that
large topology may require significant time to start. Increase
'TcpDiscoverySpi.networkTimeout' configuration property if getting this
message on the starting nodes [networkTimeout=30000]

error_log.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n6987/error_log.txt>  



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by hitendrapratap <hi...@target.com>.
Looks like its working. But we are testing with more data. Will keep you
posted



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p11579.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by vkulichenko <va...@gmail.com>.
Did you try suggestions given by Vladislav?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p11554.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by hitendrapratap <hi...@target.com>.
I am getting similar kind of issue. Did it get resolved for you? 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p11551.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hi,

If you think what dedlock there, you can increase
IGNITE_LONG_OPERATIONS_DUMP_TIMEOUT (through jvm system properties) and
networkTimeout (through Ignite configuration xml) to several minutes.

-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000

<bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="networkTimeout" value="150000" />

On Thu, Aug 18, 2016 at 12:22 PM, Jason <fq...@outlook.com> wrote:

> Thanks Vladislav.
>
> Will try to reproduce this issue again, and seems that this only happens in
> a big cluster.
>
> BTW, after the new node joins, and when it tries to do partition map
> exchange, seems that there's deadlock.
> In some nodes, "Failed to wait for partition map ..." and other nodes,
> "Failed to wait for partition eviction ...".
>
> And I've attached the detailed log for all 5 nodes and the config files,
> would you like to take a look?
> http://apache-ignite-users.70518.x6.nabble.com/Failed-to-
> wait-for-initial-partition-map-exchange-tt6252.html
>
> Thanks,
> -Jason
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-
> process-tp6987p7148.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Fail to join topology and repeat join process

Posted by Jason <fq...@outlook.com>.
Thanks Vladislav.

Will try to reproduce this issue again, and seems that this only happens in
a big cluster.

BTW, after the new node joins, and when it tries to do partition map
exchange, seems that there's deadlock.
In some nodes, "Failed to wait for partition map ..." and other nodes,
"Failed to wait for partition eviction ...".

And I've attached the detailed log for all 5 nodes and the config files,
would you like to take a look?
http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tt6252.html

Thanks,
-Jason




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p7148.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hi,

Communication unlikely influence to Discovery, because they use difference
thread pools.

Follow to steps:
1) Exclude Communication from DEBUG log level, and left in place DEBUG
level for package org.apache.ignite.spi.discovery.
2) Provide log file from a node, where a message like  "Node FAILED:
TcpDiscoveryNode..." appear first by time.
3) Provide GC log file from the node, who was joined.

After all logs will be received, we will move forward.

On Tue, Aug 16, 2016 at 12:45 PM, Jason <fq...@outlook.com> wrote:

> hi Vladislav,
>
> By scanning through the log, found that the serialization error may be the
> root cause. When the previous node of the new added node tried to send the
> messages (NodeAddedMessage/NodeAddFinishedMessage) to the new node, it
> failed, so it marked the new node as "failed" and sent an NodeFailedMessage
> to coordinator and when the coordinator received that, removed the new node
> and notified all the other nodes.
>
> I've attached the serialization error, would you like to help take a look
> at?
>
> Thanks,
> -Jason
>
> serialization_error.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7092/serialization_error.txt>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-
> process-tp6987p7092.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Fail to join topology and repeat join process

Posted by Jason <fq...@outlook.com>.
hi Vladislav,

By scanning through the log, found that the serialization error may be the
root cause. When the previous node of the new added node tried to send the
messages (NodeAddedMessage/NodeAddFinishedMessage) to the new node, it
failed, so it marked the new node as "failed" and sent an NodeFailedMessage
to coordinator and when the coordinator received that, removed the new node
and notified all the other nodes.

I've attached the serialization error, would you like to help take a look
at?

Thanks,
-Jason

serialization_error.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n7092/serialization_error.txt>  





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p7092.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by Jason <fq...@outlook.com>.
>> But data exchange and node discovery are taking place in different SPI.
I just worry about if the discovery thread has the high enough priority to
finish the join process first when the communication threads are also very
busy.

So when a new server node is joining the topology, after the coordinator
adds it to its local NodeRing, it begins to do the partition re-balance and
doesn't wait for all the server nodes to confirm the join process is
done(coordinator receives the NodeAddFinishedMessage again), right?

would you like to share some design doc or some diagram for the discovery
and communication SPI during the join process.

>> Can you provide log files from all nodes?
Will try to provide, but there're too many logs. Just because I turn on the
DEBUG for Discovery & Communication SPI. Or any suggestion on which module
to be turned on "DEBUG"?

BTW, would you like to suggest some keywords in the logs so that I can
extract some of them to ease your debugging?

Thanks,
-Jason



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p7084.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hello,

The timeout of join node to cluster is joinTimeout:

TcpDiscoverySpi
  joinTimeout

But data exchange and node discovery are taking place in different SPI.
Can you provide log files from all nodes?

On Mon, Aug 15, 2016 at 1:22 PM, Jason <fq...@outlook.com> wrote:

> hi Vladislav,
>
> BTW, the priority for all the threads of Discovery SPI is 10 (MAX_PRIORITY)
> by default, right? And for the other threads, the priority is
> NORMAL_PRIORITY by default, right?
>
> Is that possible that when the cache is with big data the discovery events
> cannot be delivered to the all the nodes in time?
>
> If I understand right:
> 1. the join process goes through: JoinRequestMessage -> NodeAddedMessage ->
> NodeAddFinishedMessage
> 2. every discovery event is sent by coordinator(oldest node) through all
> the
> server nodes one by one until it reaches coordinator again, then it goes to
> next message.
>
> If wrong, please correct.
>
> BTW, which parameter can control the time of node join? NetworkTime?
>                 long timeout = spi.netTimeout;
>
>                 long threshold = U.currentTimeMillis() + timeout;
>
>                 while (spiState == CONNECTING && timeout > 0) {
>                     try {
>                         mux.wait(timeout);
>
>                         timeout = threshold - U.currentTimeMillis();
>                     }
>                     catch (InterruptedException ignored) {
>                         Thread.currentThread().interrupt();
>
>                         throw new IgniteSpiException("Thread has been
> interrupted.");
>                     }
>                 }
>
>
> Thanks,
> -Jason
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-
> process-tp6987p7057.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Fail to join topology and repeat join process

Posted by Jason <fq...@outlook.com>.
hi Vladislav,

BTW, the priority for all the threads of Discovery SPI is 10 (MAX_PRIORITY)
by default, right? And for the other threads, the priority is
NORMAL_PRIORITY by default, right?

Is that possible that when the cache is with big data the discovery events
cannot be delivered to the all the nodes in time?

If I understand right:
1. the join process goes through: JoinRequestMessage -> NodeAddedMessage ->
NodeAddFinishedMessage
2. every discovery event is sent by coordinator(oldest node) through all the
server nodes one by one until it reaches coordinator again, then it goes to
next message.

If wrong, please correct.

BTW, which parameter can control the time of node join? NetworkTime? 
                long timeout = spi.netTimeout;

                long threshold = U.currentTimeMillis() + timeout;

                while (spiState == CONNECTING && timeout > 0) {
                    try {
                        mux.wait(timeout);

                        timeout = threshold - U.currentTimeMillis();
                    }
                    catch (InterruptedException ignored) {
                        Thread.currentThread().interrupt();

                        throw new IgniteSpiException("Thread has been
interrupted.");
                    }
                }


Thanks,
-Jason




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p7057.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by Jason <fq...@outlook.com>.
Thanks, Vladislav.

Have already tuned the GC parameters according to that link.

Seems that this error happens very frequently when the cache is very big
like over 15G per node in off-heap. 

BTW, found that in the remote nodes, the JoinEvent and FailedEvent were
received at almost the same time. Any idea about this?

[2016.08.11 00:36:34,994 PDT][INFO
][disco-event-worker-#174%null%][GridDiscoveryManager] Added new node to
topology: TcpDiscoveryNode [id=eeb076c7-2b63-431b-b53d-4acef66e99f2,
addrs=[10.183.142.50, 10.65.84.249, 127.0.0.1], sockAddrs=[/127.0.0.1:47500,
/10.183.142.50:47500, CO3SCH050520537/10.65.84.249:47500], discPort=47500,
order=1158, intOrder=662, lastExchangeTime=1470900973709, loc=false,
ver=1.7.0#20160801-sha1:383273e3, isClient=false]
[2016.08.11 00:36:34,996 PDT][INFO
][disco-event-worker-#174%null%][GridDiscoveryManager] Topology snapshot
[ver=1158, servers=20, clients=146, CPUs=1336, heap=360.0GB]
[2016.08.11 00:36:35,439 PDT][INFO
][exchange-worker-#176%null%][GridCachePartitionExchangeManager] Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1157,
minorTopVer=0], evt=NODE_FAILED, node=eeb076c7-2b63-431b-b53d-4acef66e99f2]
[2016.08.11 00:36:35,625 PDT][WARN
][disco-event-worker-#174%null%][GridDiscoveryManager] Node FAILED:
TcpDiscoveryNode [id=eeb076c7-2b63-431b-b53d-4acef66e99f2,
addrs=[10.183.142.50, 10.65.84.249, 127.0.0.1], sockAddrs=[/127.0.0.1:47500,
/10.183.142.50:47500, CO3SCH050520537/10.65.84.249:47500], discPort=47500,
order=1158, intOrder=662, lastExchangeTime=1470900973709, loc=false,
ver=1.7.0#20160801-sha1:383273e3, isClient=false]



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p7045.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by Vladislav Pyatkov <vl...@gmail.com>.
Most likely, rebalancing, after node joined to cluster, entails garbage
collection.
You can see how to configure GC in this tutorial[1].
Can you provide log from joining node and GC log from there?

[1]: https://apacheignite.readme.io/docs/jvm-and-system-tuning

On Fri, Aug 12, 2016 at 4:53 AM, Jason <fq...@outlook.com> wrote:

> The attached is the config file.
>
> For server node, one node one machine. For the client node, one machine may
> have few ones.
>
> Just do a rough check, the error logs in all the server nodes are almost
> the
> same.
>
> default-config.xml
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7002/default-config.xml>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-
> process-tp6987p7002.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Fail to join topology and repeat join process

Posted by Jason <fq...@outlook.com>.
The attached is the config file.

For server node, one node one machine. For the client node, one machine may
have few ones.

Just do a rough check, the error logs in all the server nodes are almost the
same.

default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n7002/default-config.xml>  



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p7002.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fail to join topology and repeat join process

Posted by vkulichenko <va...@gmail.com>.
Hi Jason,

Please attach your current configuration and logs from all nodes. Do you
have nodes running on the same physical box?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tp6987p6997.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.