Socket Closure due to write timout

[thammoud <th...@yahoo.com>]

Hello,

I am trying to understand a socket write timeout filling our logs. While the
jobs do finish eventually, they are a bit unsettling and would appreciate an
explanation. The easy workaround is to seriously bump up the socket write
timeout by maximum of affinity task time. However, why is the task time
related to the socket timeout? If the task takes too long the socket closes?

The example attached is an affinity call where the callable sleeps for a
time longer than the socket write timeout. You will see the client node
(TcpCommunicationSpi) closing and recreating the socket. Needlessly (IMHO).
Thank you very much for any insights. 

test.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n6391/test.zip>  



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Socket-Closure-due-to-write-timout-tp6391.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Socket Closure due to write timout

[thammoud <th...@yahoo.com>]

You are almost correct. However few things of note:

1) The error messages are unsettling and they do cause a lot of
re-transmissions from servers to clients. 
2) There was never anything wrong with the socket itself so the error is a
bit misleading. Just the jobs taking a long time to execute. There should be
nothing wrong with that.
3) I had to increase the queue size as there were queue starvation warnings
even if the socket timeout was increased.

Bottom line, I had to greatly bump up the queue size on the tcp provider and
everything worked like I expected then and only then.

Please keep up the great work. 





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Socket-Closure-due-to-write-timout-tp6391p6428.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Socket Closure due to write timout

[vkulichenko <va...@gmail.com>]

Do I understand correctly that it doesn't actually break anything? Basically,
this is true, the connection can be closed if there are no messages being
sent between nodes, but it will be recreated if needed. Under the high load
when the communication is more frequent, this should never happen.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Socket-Closure-due-to-write-timout-tp6391p6426.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Socket Closure due to write timout

[thammoud <th...@yahoo.com>]

Hi Val,

TcpCommunicationSpi.setMessageQueueLimit is the one that resolved the issue.
I increased it a lot. The timeout is not a result of an unstable network. I
do run multiple instances on the same machine. The timeout is not a result
of a socket error but a function of how long a worker takes to accomplish
the task. The example that I attached earlier clearly shows the behavior.
The lower the thread sleep in the worker, the lower the chances of getting
the error. 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Socket-Closure-due-to-write-timout-tp6391p6418.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Socket Closure due to write timout

[vkulichenko <va...@gmail.com>]

Hi,

It sounds like that your network is not very stable, so increasing
connection timeout helps to make the system more stable. This is fine, out
default settings mostly tuned for fast stable network to provide better
performance.

What queue are you referencing to?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Socket-Closure-due-to-write-timout-tp6391p6405.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Socket Closure due to write timout

[thammoud <th...@yahoo.com>]

When increase the timeout (Very large) the issue goes away, however, we start
getting queue size (Unack'd) messages. What I did was bump up the message
queue size and decreased the socket timeout and it works perfectly. 

Is bumping up the queue size an anti-pattern? We do send a lot of tasks to
be executed at once. 





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Socket-Closure-due-to-write-timout-tp6391p6392.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.