You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jackson Chung (Created) (JIRA)" <ji...@apache.org> on 2012/01/13 02:13:39 UTC

[jira] [Created] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

-Dreplace_token leaves old node (IP) in the gossip with the token.
------------------------------------------------------------------

                 Key: CASSANDRA-3736
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
             Project: Cassandra
          Issue Type: Bug
            Reporter: Jackson Chung


https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,

however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:

{noformat}
 INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
 INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
 INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
 INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
 INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
 INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
 INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
 INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
 INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
 INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
 INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
 INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
 INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
 INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
 INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
 INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
 INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
 INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
 INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
 INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
{noformat}

in the above, /50.56.58.55 was the replaced IP.

tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:

{noformat}
riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               85070591730234615865843651857942052864      
50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
{noformat}

gossipinfo:
{noformat}
$ ./bin/nodetool -h localhost gossipinfo
/50.56.58.55
  LOAD:6835.0
  SCHEMA:00000000-0000-1000-0000-000000000000
  RPC_ADDRESS:50.56.58.55
  STATUS:NORMAL,85070591730234615865843651857942052864
  RELEASE_VERSION:1.0.7-SNAPSHOT
/50.56.59.68
  LOAD:6835.0
  SCHEMA:00000000-0000-1000-0000-000000000000
  RPC_ADDRESS:50.56.59.68
  STATUS:NORMAL,60502102442797279294142560823234402248
  RELEASE_VERSION:1.0.7-SNAPSHOT
action-quick2/50.56.31.186
  LOAD:11387.0
  SCHEMA:00000000-0000-1000-0000-000000000000
  RPC_ADDRESS:50.56.31.186
  STATUS:NORMAL,85070591730234615865843651857942052864
  RELEASE_VERSION:1.0.7-SNAPSHOT
{noformat}

Note that at 1 point earlier it seems to have been removed:

$ ./bin/nodetool -h localhost gossipinfo
/50.56.59.68
  LOAD:13815.0
  SCHEMA:00000000-0000-1000-0000-000000000000
  RPC_ADDRESS:50.56.59.68
  STATUS:NORMAL,60502102442797279294142560823234402248
  RELEASE_VERSION:1.0.7-SNAPSHOT
action-quick2/50.56.31.186
  LOAD:13725.0
  SCHEMA:00000000-0000-1000-0000-000000000000
  RPC_ADDRESS:50.56.31.186
  STATUS:NORMAL,85070591730234615865843651857942052864
  RELEASE_VERSION:1.0.7-SNAPSHOT

riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster

 INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP

 INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Jackson Chung (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185398#comment-13185398 ] 

Jackson Chung commented on CASSANDRA-3736:
------------------------------------------

bq. is 50.56.58.55 up and cassandra is running?

no. the Cassandra on 50.56.58.55 was not UP/had shutdown. But the IP is available, though i don't think that matters.

so my test case was simply:
1) start 2 nodes (A , B).  With A being the seed, B bootstrap into it (by specifying a token)
2) stop B (after B had successfully joined)
3) start C with -Dcassandra.replace_token=<B's token>

continuing restarting C (without the replace_token param) could observe the behavior.
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188911#comment-13188911 ] 

Vijay commented on CASSANDRA-3736:
----------------------------------

Yes and the fix attached with this ticket will also remove the node from the System table, while replacing hence you wont even see the following message...

>>> INFO [GossipStage:1] 2012-01-18 23:45:56,412 Gossiper.java (line 800) InetAddress /50.56.58.55 is now UP

The problem is that we remove the node after 30 seconds.... Meanwhile the gossip will make the other node know about .55 and hence the message in the other node. 
The patch will fix this by removing the information from the System table in the first place instead of restart which triggering it to reappear. Can you try redoing the test? it doesn't appear back in my tests.
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.8
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams updated CASSANDRA-3736:
----------------------------------------

             Reviewer: brandon.williams
          Component/s: Core
    Affects Version/s: 1.0.0
        Fix Version/s: 1.0.7
             Assignee: Vijay
    
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "David Strauss (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185385#comment-13185385 ] 

David Strauss commented on CASSANDRA-3736:
------------------------------------------

Hi Vijay,

I filed the ticket with DataStax that prompted this issue. I'm not 100% certain whether the node we replaced was fully and consistently offline from the point we performed the replacement. I *believe* it was, especially because the -Dreplace_token refuses to work if the node being replaced is online and we took no further action to bring the replaced node back (its VM wasn't initializing any network interfaces other than "lo").

Even if the replaced node comes back, it shouldn't be allowed to re-join the ring with a token already owned by an "Up" node. It should be subjected to the same condition -Dreplace_token is, where the token being used by the new ring member must be owned by a "Down" node.

David
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185899#comment-13185899 ] 

Brandon Williams commented on CASSANDRA-3736:
---------------------------------------------

I suspect we have the same issue as I outlined in CASSANDRA-3737
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "David Strauss (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185388#comment-13185388 ] 

David Strauss commented on CASSANDRA-3736:
------------------------------------------

Alternatively, it would be good for Cassandra to provide a convenient (nodetool) way to drop the "Down" IP when a token is simultaneously occupied by one "Up" IP and at least one "Down" IP.
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Vijay (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vijay resolved CASSANDRA-3736.
------------------------------

       Resolution: Fixed
    Fix Version/s: 1.0.8

Committed both in 1.0 and trunk. Thanks!
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.8, 1.1
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185379#comment-13185379 ] 

Vijay commented on CASSANDRA-3736:
----------------------------------

Hi Jackson, Just a clarification is 50.56.58.55 up and cassandra is running?

 INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

This happens when the replaced node is running or resurrected. 
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "David Strauss (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185385#comment-13185385 ] 

David Strauss edited comment on CASSANDRA-3736 at 1/13/12 1:52 AM:
-------------------------------------------------------------------

Hi Vijay,

I filed the ticket with DataStax that prompted this issue. I'm not 100% certain whether the node we replaced was fully and consistently offline from the point we performed the replacement. I *believe* it was, especially because the -Dreplace_token refuses to work if the node being replaced is online --- and we took no further action to bring the replaced node back (its VM wasn't initializing any network interfaces other than "lo").

Even if the replaced node comes back, it shouldn't be allowed to re-join the ring with a token already owned by an "Up" node. It should be subjected to the same condition -Dreplace_token is, where the token being used by the new ring member must be owned by a "Down" node.

David
                
      was (Author: davidstrauss):
    Hi Vijay,

I filed the ticket with DataStax that prompted this issue. I'm not 100% certain whether the node we replaced was fully and consistently offline from the point we performed the replacement. I *believe* it was, especially because the -Dreplace_token refuses to work if the node being replaced is online -- and we took no further action to bring the replaced node back (its VM wasn't initializing any network interfaces other than "lo").

Even if the replaced node comes back, it shouldn't be allowed to re-join the ring with a token already owned by an "Up" node. It should be subjected to the same condition -Dreplace_token is, where the token being used by the new ring member must be owned by a "Down" node.

David
                  
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Jackson Chung (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188817#comment-13188817 ] 

Jackson Chung commented on CASSANDRA-3736:
------------------------------------------

looks like fix from CASSANDRA-3747 got the fix.

the replacement node would still get this once:
 INFO [GossipStage:1] 2012-01-18 23:45:56,412 Gossiper.java (line 834) Node /50.56.58.55 is now part of the cluster
 INFO [GossipStage:1] 2012-01-18 23:45:56,412 Gossiper.java (line 800) InetAddress /50.56.58.55 is now UP
 INFO [GossipStage:1] 2012-01-18 23:45:56,413 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
 INFO [GossipTasks:1] 2012-01-18 23:46:05,805 Gossiper.java (line 814) InetAddress /50.56.58.55 is now dead.
 INFO [GossipTasks:1] 2012-01-18 23:46:26,819 Gossiper.java (line 628) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip

but its quiet after that.

the other node would receive the same info also:

 INFO [GossipTasks:1] 2012-01-18 23:45:57,486 Gossiper.java (line 628) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip

and the gossipinfo of those nodes are the matching:


$ ./bin/nodetool -h 50.56.31.186 gossipinfo
/50.56.59.68
  RELEASE_VERSION:1.0.7-SNAPSHOT
  LOAD:6820.0
  RPC_ADDRESS:50.56.59.68
  STATUS:NORMAL,0
  SCHEMA:00000000-0000-1000-0000-000000000000
action-quick2/50.56.31.186
  RELEASE_VERSION:1.0.7-SNAPSHOT
  RPC_ADDRESS:50.56.31.186
  STATUS:NORMAL,85070591730234615865843651857942052864
  LOAD:11372.0
  SCHEMA:00000000-0000-1000-0000-000000000000

$ ./bin/nodetool -h 50.56.59.68 gossipinfo
action-quick/50.56.59.68
  SCHEMA:00000000-0000-1000-0000-000000000000
  RELEASE_VERSION:1.0.7-SNAPSHOT
  LOAD:6820.0
  RPC_ADDRESS:50.56.59.68
  STATUS:NORMAL,0
/50.56.31.186
  SCHEMA:00000000-0000-1000-0000-000000000000
  RELEASE_VERSION:1.0.7-SNAPSHOT
  LOAD:11372.0
  RPC_ADDRESS:50.56.31.186
  STATUS:NORMAL,85070591730234615865843651857942052864

                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.8
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189302#comment-13189302 ] 

Brandon Williams commented on CASSANDRA-3736:
---------------------------------------------

If CASSANDRA-3747 solved this, then I don't think there's any full solution here worth applying, since this is mostly just a cosmetic problem and not worth introducing a possibly destabilizing change over.  Anyone running into this can use CASSANDRA-3337 to remove it, or avoid replacing tokens.

+1 to this patch for 1.0 and trunk, though.
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.1
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vijay updated CASSANDRA-3736:
-----------------------------

    Attachment: 0001-CASSANDRA-3736.patch

Simple patch to remove from SYSTEM_TABLE/RING_KEY when token is replaced.
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Tamar Fraenkel (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496136#comment-13496136 ] 

Tamar Fraenkel edited comment on CASSANDRA-3736 at 11/13/12 12:32 PM:
----------------------------------------------------------------------

I have the same issue with Cassandra 1.0.11 (used DataStax AMI).
I thought it was supposed to be solved already.
I see those messages on the node that was started using -Dcassandra.replace_token=<token>.

>From time to time I also see 
{color:blue} 
 INFO [GossipTasks:1] 2012-11-13 12:26:38,195 Gossiper.java (line 818) InetAddress /<dead_node_ip> is now dead.
 INFO [GossipTasks:1] 2012-11-13 12:26:58,203 Gossiper.java (line 632) FatClient /<dead_node_ip> has been silent for 30000ms, removing from gossip
 INFO [GossipStage:1] 2012-11-13 12:27:59,210 Gossiper.java (line 838) Node /<dead_node_ip> is now part of the cluster
 INFO [GossipStage:1] 2012-11-13 12:27:59,210 Gossiper.java (line 804) InetAddress /<dead_node_ip> is now UP
 INFO [GossipStage:1] 2012-11-13 12:27:59,210 StorageService.java (line 1017) Nodes /<dead_node_ip> and /<replacing_node_ip> have the same token 113427455640312821154458202477256070484.  Ignoring /<dead_node_ip>

{color}


                
      was (Author: tamarfraenkel):
    I have the same issue with Cassandra 1.0.11 (used DataStax AMI).
I thought it was supposed to be solved already.
I see those messages on the node that was started using -Dcassandra.replace_token=<token>.


                  
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.8, 1.1.0
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Jackson Chung (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185879#comment-13185879 ] 

Jackson Chung commented on CASSANDRA-3736:
------------------------------------------

fix no good.

and to ensure fix is deployed, checked the compiled class:

$ javap -c -private -classpath ./build/classes/main/ org.apache.cassandra.db.SystemTable | grep "updateToken(java.net.Inet" -A10
public static synchronized void updateToken(java.net.InetAddress, org.apache.cassandra.dht.Token);
  Code:
   0:   aload_0
   1:   invokestatic    #51; //Method org/apache/cassandra/utils/FBUtilities.getLocalAddress:()Ljava/net/InetAddress;
   4:   if_acmpne       12
   7:   aload_1
   8:   invokestatic    #52; //Method removeToken:(Lorg/apache/cassandra/dht/Token;)V
   11:  return

to ensure removeToken is added (per the patch)

and the classpath of the jvm is using it:

 INFO 20:32:57,083 Classpath: ./bin/../conf:./bin/*../build/classes/main*:./bin/../build/classes/thrift:./bin/../lib/antlr-3.2.jar:./bin/../lib/avro-1.4.0-fixes.jar:./bin/../lib/avro-1.4.0-sources-fixes.jar:./bin/../lib/commons-cli-1.1.jar:./bin/../lib/commons-codec-1.2.jar:./bin/../lib/commons-lang-2.4.jar:./bin/../lib/compress-lzf-0.8.4.jar:./bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:./bin/../lib/guava-r08.jar:./bin/../lib/high-scale-lib-1.1.2.jar:./bin/../lib/jackson-core-asl-1.4.0.jar:./bin/../lib/jackson-mapper-asl-1.4.0.jar:./bin/../lib/jamm-0.2.5.jar:./bin/../lib/jline-0.9.94.jar:./bin/../lib/json-simple-1.1.jar:./bin/../lib/libthrift-0.6.jar:./bin/../lib/log4j-1.2.16.jar:./bin/../lib/servlet-api-2.5-20081211.jar:./bin/../lib/slf4j-api-1.6.1.jar:./bin/../lib/slf4j-log4j12-1.6.1.jar:./bin/../lib/snakeyaml-1.6.jar:./bin/../lib/snappy-java-1.0.4.1.jar:./bin/../lib/jamm-0.2.5.jar

log from the replacement node:
{noformat}
 INFO 20:34:27,856 Listening for thrift clients...
 INFO 20:35:28,750 Node /50.56.58.55 is now part of the cluster
 INFO 20:35:28,750 InetAddress /50.56.58.55 is now UP
 INFO 20:35:28,751 Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
 INFO 20:35:38,841 InetAddress /50.56.58.55 is now dead.
 INFO 20:35:58,852 FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
 INFO 20:36:59,786 Node /50.56.58.55 is now part of the cluster
 INFO 20:36:59,787 InetAddress /50.56.58.55 is now UP
 INFO 20:36:59,787 Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
 INFO 20:37:09,887 InetAddress /50.56.58.55 is now dead.
 INFO 20:37:29,898 FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
{noformat}


                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185389#comment-13185389 ] 

Brandon Williams commented on CASSANDRA-3736:
---------------------------------------------

bq. Alternatively, it would be good for Cassandra to provide a convenient (nodetool) way to drop the "Down" IP when a token is simultaneously occupied by one "Up" IP and at least one "Down" IP.

CASSANDRA-3337 is designed to handle these kinds of situations (where gossip is not doing the right thing naturally)
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams updated CASSANDRA-3736:
----------------------------------------

    Fix Version/s:     (was: 1.0.8)
                   1.1
    
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.1
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "David Strauss (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185385#comment-13185385 ] 

David Strauss edited comment on CASSANDRA-3736 at 1/13/12 1:52 AM:
-------------------------------------------------------------------

Hi Vijay,

I filed the ticket with DataStax that prompted this issue. I'm not 100% certain whether the node we replaced was fully and consistently offline from the point we performed the replacement. I *believe* it was, especially because the -Dreplace_token refuses to work if the node being replaced is online -- and we took no further action to bring the replaced node back (its VM wasn't initializing any network interfaces other than "lo").

Even if the replaced node comes back, it shouldn't be allowed to re-join the ring with a token already owned by an "Up" node. It should be subjected to the same condition -Dreplace_token is, where the token being used by the new ring member must be owned by a "Down" node.

David
                
      was (Author: davidstrauss):
    Hi Vijay,

I filed the ticket with DataStax that prompted this issue. I'm not 100% certain whether the node we replaced was fully and consistently offline from the point we performed the replacement. I *believe* it was, especially because the -Dreplace_token refuses to work if the node being replaced is online and we took no further action to bring the replaced node back (its VM wasn't initializing any network interfaces other than "lo").

Even if the replaced node comes back, it shouldn't be allowed to re-join the ring with a token already owned by an "Up" node. It should be subjected to the same condition -Dreplace_token is, where the token being used by the new ring member must be owned by a "Down" node.

David
                  
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.7
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.

Posted by "Tamar Fraenkel (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496136#comment-13496136 ] 

Tamar Fraenkel commented on CASSANDRA-3736:
-------------------------------------------

I have the same issue with Cassandra 1.0.11 (used DataStax AMI).
I thought it was supposed to be solved already.
I see those messages on the node that was started using -Dcassandra.replace_token=<token>.


                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.8, 1.1.0
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68 state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55 has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55 is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                85070591730234615865843651857942052864      
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248      
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864 
> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55 is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55 and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.  Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira