Re: CPP client: high latency for large messages

[davidmc <ar...@gmail.com>]

Hi again Ben,

Actually, 802 bytes is not a big message in this problem :) It's rather
small.
I did a rather extensive test in this subject... sadly I am not a TCP
implementation expert, so I cannot give a good explanation. But my results
were more or less like this:
-big messages (more than 8KB-16KB depending on the situations -> no 40 ms
latency, probably because buffer is full with 1 message and TCP decides to
send it as a packet right away)
-small messages (less than 8KB-16KB) -> 40 ms latency.
-very small messages (some hundres bytes) -> no 40 ms latency. Why? I have
no idea.

Also, I'm happy that my advice helped you :) Just a little warning... even
if the fix to the connection URL dropped your AVERAGE latency, you still may
get some 40ms latencies here and there because the broker's sockets don't
have TCP_NODELAY enabled.
So, if all you care is the average, then you should not worry anymore. If
you don't want to have any 40 ms latencies at all, then I recommend you to
slightly modify your test program to check that you don't get any latencies
in the 38-41 ms range. Also, if you do get some 40 ms latencies, think that
during those 40ms no other messages can be sent... so actually you can have
a situation similar to this:
during 1 second, 1000 messages are sent, but 20 of them have 40ms latencies.
20*40 = 800 ms, so 80% of the time is lost because of this. Yet the average
latency is 800 / 1000 = 0.8 ms (supposing the other 980 messages have
negligeable latency), kind of very low, and you may not notice that
something wrong is happening.

Well, anyway... most probably you don't need to worry any more, so I'm happy
that you got your problem solved.

David


bwatrous wrote:
> 
> Hi David,
>    Thank you for the advice.  I didn't think the tcpNoDelay would have an
> effect since I was looking at relatively large messages :)
>    Simply changing the connection URL solved the immediate problem, send
> latency dropped right down into the microseconds as expected.  So far I
> haven't had to make the suggested changes to the broker.
> 
> Ben
> 
> 
> davidmc wrote:
>> 
>> Hi there bwatrous,
>> 
>> Your average sent time for the problematic RUN 1 (40ms) indicates that
>> your problem is most probably caused by the bad Nagle's algorithm /
>> Delayed ACK timeout interaction. This timeout is 40ms in Linux and 200ms
>> in Windows / other systems.
>> 
>> In short, the solution is to enable TCP_NODELAY flag on all your sockets.
>> -This is possible in the ActiveMQ CPP client since version 2.0.1. You
>> should append tcpNoDelay=true to the connection URL.
>> -On the broker's side, you should enable the TCP_NODELAY flag in the
>> broker's sockets. For this, following the documentation, you should
>> append 'wireFormat.tcpNoDelayEnabled=true' to your connector's URI.
>> In fact, this doesn't work and it is silently ignored. I discovered that
>> 'socket.tcpNoDelay=true' works if you append it to an ActiveMQ Java
>> CLIENT's connection URI; but it still doesn't work for a BROKER's
>> connector URI.
>> So, the only solution is to go to the broker's source code
>> (org.apache.activemq.transport.tcp.TcpTransport.initialiseSocket method)
>> and add the following line of Java code:
>> --->          sock.setTcpNoDelay(true);         <---
>> at the end of the method, so that all broker's sockets are created with
>> TCP_NODELAY=true by default.
>> 
>> The parsing of options in a broker's activemq.xml connector URI is broken
>> right now, and it sadly doesn't seem to be a priority that this is fixed
>> for next releases.
>> So, I wonder how many folks encounter this '40 ms' problem, and when they
>> finally guess that the solution is the TCP_NODELAY flag (not so easy),
>> they try it and think it was not the right solution because
>> 'wireFormat.tcpNoDelayEnabled=true' or 'tcpNoDelay=true' or
>> 'socket.tcpNoDelay=true' get all silently ignored (no error message, but
>> no effect on the sockets). Probably lots of frustration and lost hours...
>> :(
>> 
>> Related messages and JIRA's:
>> https://issues.apache.org/activemq/browse/AMQ-1156
>> https://issues.apache.org/activemq/browse/AMQ-1137
>> http://issues.apache.org/activemq/browse/AMQ-1233
>> http://www.nabble.com/AMQCPP-Openwire-much-slower-than-Stomp--tf3599986s2354.html#a10055544
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/CPP-client%3A-high-latency-for-large-messages-tf3995387s2354.html#a11387205
Sent from the ActiveMQ - User mailing list archive at Nabble.com.