[jira] [Commented] (CASSANDRA-3569) Failure detector downs should not break streams

["Sylvain Lebresne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13956208#comment-13956208 ] 

Sylvain Lebresne commented on CASSANDRA-3569:
---------------------------------------------

bq. other than configuring keepalive on streaming which looks to be a pretty trivial change

Setting keep-alive on the connection is trivial, but the default keep alive time on most system (2 hours on linux I believe, or at least on my box) is way too long to be acceptable here. And to the best of my knowledge, we can't change the keep alive time from Java. So if we are to start relying on keep alive, we'll have to change the default keep alive time in some way, being it a call to sysctl (and whatever is equivalent on windows) in the startup script or some JNA call.

bq.  are we wanting to remove FD from the streaming process and rely on the underlying TCP mechanisms to handle that for us

Provided we do deal with the keep alive default timeout as described above, I'm personally fine with relying on that and removing the FD use for streaming (but as said above, regarding repairs, we'll need to keep it for the initial merkle tree computation phase).

> Failure detector downs should not break streams
> -----------------------------------------------
>
>                 Key: CASSANDRA-3569
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3569
>             Project: Cassandra
>          Issue Type: New Feature
>            Reporter: Peter Schuller
>            Assignee: Joshua McKenzie
>             Fix For: 2.1 beta3
>
>         Attachments: 3569-2.0.txt
>
>
> CASSANDRA-2433 introduced this behavior just to get repairs to don't sit there waiting forever. In my opinion the correct fix to that problem is to use TCP keep alive. Unfortunately the TCP keep alive period is insanely high by default on a modern Linux, so just doing that is not entirely good either.
> But using the failure detector seems non-sensicle to me. We have a communication method which is the TCP transport, that we know is used for long-running processes that you don't want to incorrectly be killed for no good reason, and we are using a failure detector tuned to detecting when not to send real-time sensitive request to nodes in order to actively kill a working connection.
> So, rather than add complexity with protocol based ping/pongs and such, I propose that we simply just use TCP keep alive for streaming connections and instruct operators of production clusters to tweak net.ipv4.tcp_keepalive_{probes,intvl} as appropriate (or whatever equivalent on their OS).
> I can submit the patch. Awaiting opinions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)