[jira] [Commented] (CASSANDRA-7307) New nodes mark dead nodes as up for 10 minutes

["Brandon Williams (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010403#comment-14010403 ] 

Brandon Williams commented on CASSANDRA-7307:
---------------------------------------------

I was just looking at this.. here are my results:

{noformat}
 INFO 21:58:45,820 Node /10.208.8.63 state jump to normal
 INFO 22:07:58,880 InetAddress /10.208.8.63 is now DOWN
{noformat}

I'm 100% sure the cause of this is CASSANDRA-6385.  Luckily, after CASSANDRA-4375 you can override this.  Unluckily, you have to apply a two liner from CASSANDRA-6751 because of stupid static initialization.  We should, perhaps, revisit our motivation for CASSANDRA-6385 and make the default much lower (for the common case) and let people who are using 1000 node clusters with vnodes perform the override instead.

> New nodes mark dead nodes as up for 10 minutes
> ----------------------------------------------
>
>                 Key: CASSANDRA-7307
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7307
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Richard Low
>            Assignee: Brandon Williams
>             Fix For: 1.2.17
>
>
> When doing a node replacement when other nodes are down we see the down nodes marked as up for about 10 minutes. This means requests are routed to the dead nodes causing timeouts. It also means replacing a node when multiple nodes from a replica set is extremely difficult - the node usually tries to stream from a dead node and the replacement fails.
> This isn't limited to host replacement. I did a simple test:
> 1. Create a 2 node cluster
> 2. Kill node 2
> 3. Start a 3rd node with a unique token (I used auto_bootstrap=false but I don't think this is significant)
> The 3rd node lists node 2 (127.0.0.2) as up for almost 10 minutes:
> {code}
> INFO [main] 2014-05-27 14:28:24,753 CassandraDaemon.java (line 119) Logging initialized
> INFO [GossipStage:1] 2014-05-27 14:28:31,492 Gossiper.java (line 843) Node /127.0.0.2 is now part of the cluster
> INFO [GossipStage:1] 2014-05-27 14:28:31,495 Gossiper.java (line 809) InetAddress /127.0.0.2 is now UP
> INFO [GossipTasks:1] 2014-05-27 14:37:44,526 Gossiper.java (line 823) InetAddress /127.0.0.2 is now DOWN
> {code}
> I reproduced on 1.2.15 and 1.2.16.



--
This message was sent by Atlassian JIRA
(v6.2#6252)