You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by hitendrapratap <hi...@target.com> on 2017/01/21 22:12:51 UTC
Timed out waiting for message delivery receipt
I am trying to do POC with 4 Ignite server and 2 clients. But getting the
below error on servers.
{"@timestamp":"2017-01-21T16:51:27.168-05:00","@version":1,"message":"Timed
out waiting for message delivery receipt (most probably, the reason is in
long GC pauses on remote node; consider tuning GC and increasing
'ackTimeout' configuration property). Will retry to send message with
increased timeout. Current timeout:
9990.","logger_name":"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi","thread_name":"tcp-disco-msg-worker-#2%null%","level":"WARN","level_value":30000,"HOSTNAME":""}
{"@timestamp":"2017-01-21T16:51:27.182-05:00","@version":1,"message":"Node
is out of topology (probably, due to short-time network
problems).","logger_name":"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi","thread_name":"tcp-disco-msg-worker-#2%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
{"@timestamp":"2017-01-21T16:51:27.183-05:00","@version":1,"message":"Local
node SEGMENTED: TcpDiscoveryNode [id=89420106-c961-4286-8c4d-853e4c54af04,
addrs=[127.0.0.1,xxx], sockAddrs=[xxx:8062, /127.0.0.1:8062], discPort=8062,
order=16, intOrder=9, lastExchangeTime=1485035487180, loc=true,
ver=1.8.0#20161205-sha1:9ca40dbe,
isClient=false]","logger_name":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","thread_name":"disco-event-worker-#17%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
{"@timestamp":"2017-01-21T16:51:29.912-05:00","@version":1,"message":"Stopping
local node according to configured segmentation
policy.","logger_name":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","thread_name":"disco-event-worker-#17%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
{"@timestamp":"2017-01-21T16:51:29.912-05:00","@version":1,"message":"Node
FAILED: TcpDiscoveryNode [id=1e3bc27f-6ee8-48a0-8843-d803fcf054fd,
addrs=[127.0.0.1, xxx], sockAddrs=[xxx:8062, /xxx:8062, /127.0.0.1:8062],
discPort=8062, order=3, intOrder=3, lastExchangeTime=1485035338259,
loc=false, ver=1.8.0#20161205-sha1:9ca40dbe,
isClient=false]","logger_name":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","thread_name":"disco-event-worker-#17%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Timed out waiting for message delivery receipt
Posted by luqmanahmad <lu...@gmail.com>.
See [1] for free network segmentation plugin
[1] https://github.com/luqmanahmad/ignite-plugins
<https://github.com/luqmanahmad/ignite-plugins>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Timed out waiting for message delivery receipt
Posted by hitendrapratap <hi...@target.com>.
Openstack enviornment. I am using Ignite 1.8 version and as an embedded
within the application. I did validated iptables and ports connectivity but
everything is good.
Below are some more messages:
1.Failed to send message to next node
2. Local node has detected failed nodes and started cluster-wide procedure.
To speed up failure detection please see 'Failure Detection' section under
javadoc for 'TcpDiscoverySpi'
-> I tried increasing this too. But didn't helped.
3."Node FAILED: TcpDiscoveryNode
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171p10204.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Timed out waiting for message delivery receipt
Posted by hitendrapratap <hi...@target.com>.
Please find the server and client configuration and let me know if you find
anything wrong.
*Server Configuration*:
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setConnectorConfiguration(null);
TcpCommunicationSpi comm = new TcpCommunicationSpi();
comm.setLocalPort(47100);
igniteConfiguration.setCommunicationSpi(comm);
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setLocalPort(47500);
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses("SERVER1_IP,SERVER2_IP,SERVER3_IP,SERVER4_IP");
spi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(spi);
igniteConfiguration.setTimeServerPortBase(31100);
igniteConfiguration.setFailureDetectionTimeout(60000);
*Client Configuration*:
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setClientMode(true);
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
discoverySpi.setLocalPort(47500);
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses("SERVER1_IP,SERVER2_IP,SERVER3_IP,SERVER4_IP");
discoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(discoverySpi);
igniteConfiguration.setFailureDetectionTimeout(60000);
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171p10208.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Timed out waiting for message delivery receipt
Posted by Yakov Zhdanov <yz...@apache.org>.
Very good! Please keep us up to date.
--Yakov
2017-01-24 9:30 GMT+03:00 hitendrapratap <
hitendrapratapsingh.singh@target.com>:
> I has been resolved when increased "ignite.failure.detection.timeout" to
> 60sec from 10sec.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-
> receipt-tp10171p10209.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Timed out waiting for message delivery receipt
Posted by hitendrapratap <hi...@target.com>.
I has been resolved when increased "ignite.failure.detection.timeout" to
60sec from 10sec.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171p10209.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Timed out waiting for message delivery receipt
Posted by Yakov Zhdanov <yz...@apache.org>.
Hello!
What environment do you use? Can you please make sure you do not have any
firewall/connectivity/iptables issues?
--Yakov
Re: Timed out waiting for message delivery receipt
Posted by hitendrapratap <hi...@target.com>.
One more thing which I have noticed - First I have started 4 servers and they
were able to form a cluster of 4 nodes. But when I have started 1 client
then only 2 servers were able to communicate with the client. And other were
not able to communicate with client
Below were the messages on the servers which were not able to communicate
with the client:
Node is out of topology (probably, due to short-time network problems).
Local node SEGMENTED: TcpDiscoveryNode
Stopping local node according to configured segmentation policy.
Node FAILED: TcpDiscoveryNode
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171p10189.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Timed out waiting for message delivery receipt
Posted by hitendrapratap <hi...@target.com>.
I was getting the errors while starting the cluster and there is no data in
the cache.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171p10188.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Timed out waiting for message delivery receipt
Posted by Denis Magda <dm...@apache.org>.
Hi,
The first thing I would check is that there are no long stop-the-world GC pauses in your cluster. Get to know this JVM and system tuning page:
https://apacheignite.readme.io/docs/jvm-and-system-tuning <https://apacheignite.readme.io/docs/jvm-and-system-tuning>
—
Denis
> On Jan 21, 2017, at 2:12 PM, hitendrapratap <hi...@target.com> wrote:
>
> I am trying to do POC with 4 Ignite server and 2 clients. But getting the
> below error on servers.
>
>
> {"@timestamp":"2017-01-21T16:51:27.168-05:00","@version":1,"message":"Timed
> out waiting for message delivery receipt (most probably, the reason is in
> long GC pauses on remote node; consider tuning GC and increasing
> 'ackTimeout' configuration property). Will retry to send message with
> increased timeout. Current timeout:
> 9990.","logger_name":"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi","thread_name":"tcp-disco-msg-worker-#2%null%","level":"WARN","level_value":30000,"HOSTNAME":""}
> {"@timestamp":"2017-01-21T16:51:27.182-05:00","@version":1,"message":"Node
> is out of topology (probably, due to short-time network
> problems).","logger_name":"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi","thread_name":"tcp-disco-msg-worker-#2%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
> {"@timestamp":"2017-01-21T16:51:27.183-05:00","@version":1,"message":"Local
> node SEGMENTED: TcpDiscoveryNode [id=89420106-c961-4286-8c4d-853e4c54af04,
> addrs=[127.0.0.1,xxx], sockAddrs=[xxx:8062, /127.0.0.1:8062], discPort=8062,
> order=16, intOrder=9, lastExchangeTime=1485035487180, loc=true,
> ver=1.8.0#20161205-sha1:9ca40dbe,
> isClient=false]","logger_name":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","thread_name":"disco-event-worker-#17%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
> {"@timestamp":"2017-01-21T16:51:29.912-05:00","@version":1,"message":"Stopping
> local node according to configured segmentation
> policy.","logger_name":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","thread_name":"disco-event-worker-#17%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
> {"@timestamp":"2017-01-21T16:51:29.912-05:00","@version":1,"message":"Node
> FAILED: TcpDiscoveryNode [id=1e3bc27f-6ee8-48a0-8843-d803fcf054fd,
> addrs=[127.0.0.1, xxx], sockAddrs=[xxx:8062, /xxx:8062, /127.0.0.1:8062],
> discPort=8062, order=3, intOrder=3, lastExchangeTime=1485035338259,
> loc=false, ver=1.8.0#20161205-sha1:9ca40dbe,
> isClient=false]","logger_name":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","thread_name":"disco-event-worker-#17%null%","level":"WARN","level_value":30000,"HOSTNAME":"xxx"}
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Timed-out-waiting-for-message-delivery-receipt-tp10171.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.