OutboundMessagingConnection behavior under high latencies between Cassandra nodes

[Enrique Saurez <en...@gmail.com>]

Hi!

I have a design question about the communication layer in Cassandra. 
Some context first:

 1.      I am using Cassandra in a high inter-node communication latency
    environment, e.g latency between nodes is higher than 60 ms, like in
    multi-datacenter environments.
 2.      Currently, I am running my tests in a three node cluster, with
    replication factor equal to three.
 3.      I am working in a low throughput scenario, where I am trying to
    reduce the latency as much as possible for quorum operations (both
    reads and writes)
 4.      All my query-related messages between nodes can be considered
    small (all query-related messages would most likely go through the
    smallMessageChannel).

I am trying to understand some of the design decisions made on the 
org.apache.cassandra.net.async.OutboundMessagingConnection and 
org.apache.cassandra.net.async.OutboundMessagingPool.

If I disable completely coalescing (e.g. using otc_coalescing_strategy 
as disabled), and I send two almost simultaneous request that affects 
the same set of replicas, after the coordinator sends the message to one 
of the data nodes. In the data node, the fastest query would complete 
and use the OutboundMessagingConnection and block until a TCP ACK is 
received, then all the other pending messages in the data node would be 
sent in the next TCP PSH. I notice this by running tcpdump and Wireshark 
in the nodes and analyzing the TCP connections between nodes.

My question is: Why is Cassandra using TCP in a synchronous manner? and 
not using some type of windowing to allow more messages to go in the 
wire before waiting for an ack?. Is this because of some of the 
guarantees that Cassandra provides?

I know coalescing could help, but if the messages are separated by more 
than the window (200 us by default in the fix setting), then it would 
need to wait for a whole RTT (60 ms), before it is sent, so for many 
operations in my system, I see that the latency perceived is on average 
2 RTT instead of the quorum round trip that should be the slowest of all 
the communication paths.

I don't know enough about Netty and TCP to figure out this by myself, 
but I think I understand the basics of OutboundMessagingConnection and 
OutboundMessagingConnectionPool, so I am not sure why that requirement 
for synchronous TCP PSH and ACK is needed.

Thanks a lot for your help! Cassandra code is really good, and I was 
able to understand many of the components thanks to it.

Best regards,
Enrique Saurez