[jira] [Assigned] (CASSANDRA-6615) Changing the IP of a node on a live cluster leaves gossip infos and throws Exceptions

["Brandon Williams (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams reassigned CASSANDRA-6615:
-------------------------------------------

    Assignee: Tyler Hobbs

What is happening is that the token moves correctly on the other nodes, but the node that moved gets the host ID collision adding the old ip which is still in gossip, with its own host ID.  Because of this, it never removes the old ip, and thus re-propagates it to the other nodes after their quarantine expires, causing them to now get the same collision.

> Changing the IP of a node on a live cluster leaves gossip infos and throws Exceptions
> -------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6615
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6615
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Fabien Rousseau
>            Assignee: Tyler Hobbs
>
> Following this procedure : https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/  to change the IP of a node, we encountered an issue :
>  - logs contains: "java.lang.RuntimeException: Host ID collision between active endpoint /127.0.0.5 and /127.0.0.3"
>  - logs also indicate that the old IP is being removed of the cluster (FatClient timeout), then added again...
>  - nodetool gossipinfo still list old IP (even a few hours after...)
>  - the old IP is still seen as "UP" in the cluster... (according to the logs...)
> Below is a small shell script which allows to reproduce the scenario...
> {noformat}
> #! /bin/bash
> CLUSTER=$1
> ccm create $CLUSTER --cassandra-dir=.
> ccm populate -n 2
> ccm start
> ccm add node3 -i 127.0.0.3 -j 7300 -b
> ccm node3 start
> ccm node3 ring
> ccm node3 stop
> sed -i 's/127.0.0.3/127.0.0.5/g' ~/.ccm/$CLUSTER/node3/node.conf 
> sed -i 's/127.0.0.3/127.0.0.5/g' ~/.ccm/$CLUSTER/node3/conf/cassandra.yaml
> ccm node3 start
> sleep 3
> nodetool --host 127.0.0.5 --port 7300 gossipinfo
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)