[jira] [Comment Edited] (CASSANDRA-9237) Gossip messages subject to head of line blocking by other intra-cluster traffic

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517288#comment-14517288 ] 

Benedict edited comment on CASSANDRA-9237 at 4/28/15 4:05 PM:
--------------------------------------------------------------

TCP connections are generally designed to share bandwidth fairly, so having an extra connection means a low traffic high priority connection will be more likely to see its messages arrive promptly. Either way, having multiple connections means that if one gets backed up you can at least hop to the front of the application level queues that are now backed up.

I agree that extra threads are unacceptable, but having a single thread serving a collection of NIO sockets to each host for Gossip, as I suggested in CASSANDRA-8789 seems like it has no major downsides besides the work involved. Ideally we would precede such work by isolating the transport implementation of MessagingService from its utilisation (and permit different messages go over different transports), so that we can drop in a multitude of different schemes (OIO, NIO, UDP, SCTP, whatever), and then the problem becomes very tractable. I would argue it becomes very sensible, even.

Constraining huge messages to such an NIO socket also makes sense to me, since I doubt the throughput problems we see with NIO are a major issue for large messages, and we don't really want lots of unnecessary threads hanging around. This also makes it easier to transparently switch implementation based on e.g. cluster size, or frequency of communication (non-vnode clusters could have OIO connections to their direct peers, and NIO to everyone else, for instance).


was (Author: benedict):
TCP connections are generally designed to share bandwidth fairly, so having an extra connection means a low traffic high priority connection will be more likely to see its messages arrive promptly. Either way, having multiple connections means that if one gets backed up you can at least hop to the front of the application level queues that are now backed up.

I agree that extra threads are unacceptable, but having a single thread serving a collection of NIO sockets to each host for Gossip, as I suggested in CASSANDRA-8789 seems like it has no major downsides besides the work involved. Ideally we would precede such work by isolating the transport implementation of MessagingService from its utilisation (and permit different messages go over different transports), so that we can drop in a multitude of different schemes (OIO, NIO, UDP, SCTP, whatever), and then the problem becomes very tractable. I would argue it becomes very sensible, even.

> Gossip messages subject to head of line blocking by other intra-cluster traffic
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-9237
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9237
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>
> Reported as an issue over less than perfect networks like VPNs between data centers.
> Gossip goes over the small message socket where small is 64k which isn't particularly small. This is done for performance to keep most traffic on one hot socket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)