You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Karl Mueller (JIRA)" <ji...@apache.org> on 2012/06/16 00:41:42 UTC
[jira] [Created] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Karl Mueller created CASSANDRA-4347:
---------------------------------------
Summary: IP change of node requires assassinate to really remove old IP
Key: CASSANDRA-4347
URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.10
Environment: RHEL6, 64bit
Reporter: Karl Mueller
Priority: Minor
In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Harpreet Kaur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496384#comment-13496384 ]
Harpreet Kaur commented on CASSANDRA-4347:
------------------------------------------
I am on vacation till 8/17. For ERP DBA related issues contact Neeren at 650 279-6155. For escalations contact Kevin Railsback.
Thanks,
Harpreet Kaur
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: 4347.txt, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt, LocationInfo-hd-279-Data.db
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401682#comment-13401682 ]
Brandon Williams commented on CASSANDRA-4347:
---------------------------------------------
Nope, I have my own now :)
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401700#comment-13401700 ]
Karl Mueller commented on CASSANDRA-4347:
-----------------------------------------
Actually, this morning I started to see the same messages, approximately 3 days later..
Related to https://issues.apache.org/jira/browse/CASSANDRA-2961 somehow? Some people on IRC thought so, maybe.
Assassinate is NOT removing them successfully, anymore.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401681#comment-13401681 ]
Karl Mueller commented on CASSANDRA-4347:
-----------------------------------------
Brandon,
Since you can reproduce, do you still want the logs? I think I still have them if needed.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Mueller updated CASSANDRA-4347:
------------------------------------
Attachment: dev-cass-post-assassinate-gossipinfo.txt
This file contains gossipinfo from the 3-node cluster we already moved, after assassinate has run on each node for its own old IP.
The new IPs are all 10.93.15.xx and the old IPs are all 10.12.x.x.
The old IPs are as follows:
dev-cass00 - 10.12.9.160
dev-cass01 - 10.12.9.157
dev-cass02 - 10.12.9.33
I believe dev-cass00 has restarted since the assinate, but the others haven't.
New IPs are:
dev-cass00 - 10.93.15.10
dev-cass01 - 10.93.15.11
dev-cass02 - 10.93.15.12
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Priority: Minor
> Attachments: dev-cass-post-assassinate-gossipinfo.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396215#comment-13396215 ]
Karl Mueller commented on CASSANDRA-4347:
-----------------------------------------
You mean, before I did the assassinate? All of the nodes at this point are post-assassinate. I'm attaching the gossipinfo from the 3-node cluster in the current state which is showing some old IPs.
(I thought assassinate went cross-cluster?)
I'm moving another cluster this week, and I'll try to grab a gossipinfo and the system tables during transition from that set. I expect it will have the same issues.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Priority: Minor
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Mueller updated CASSANDRA-4347:
------------------------------------
Attachment: LocationInfo-hd-279-Data.db
LocationInfo file attached from after node is re-IP'd and rejoins the cluster. This is in the problem state.
I also have system snapshots of before the move and after the assassinate, as well as a node that isn't moving (same snapshots) if you want them.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396232#comment-13396232 ]
Brandon Williams commented on CASSANDRA-4347:
---------------------------------------------
Yes, a pre-assassinate capture is what we need. The old IPs showing in the LEFT state is assassinate working (and they do appear cross-cluster)
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: dev-cass-post-assassinate-gossipinfo.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-4347:
-------------------------------------------
Assignee: Brandon Williams
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: dev-cass-post-assassinate-gossipinfo.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396251#comment-13396251 ]
Karl Mueller commented on CASSANDRA-4347:
-----------------------------------------
OK, I'll grab one this week when we do the move.
I assume you want the LocationInfo CF, or do you want the entire system keyspace?
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: dev-cass-post-assassinate-gossipinfo.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401685#comment-13401685 ]
Karl Mueller commented on CASSANDRA-4347:
-----------------------------------------
Mine log's more special! ;) just kidding.
My opinion on the urgency of the bug would depend on how long 1.0.x will be around. It's sort of an annoying yet in-your-face type of bug that doesn't really seem to have a "problem" beyond creating a lot of bad log entries. Yet I could see people running into it, and then having to find the workaround.
Perhaps in the interim some type of log message could simply be added about maybe trying assassinate? It should be easy to see "Oh there's two IPs for this one token. Is one old?"
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401471#comment-13401471 ]
Brandon Williams commented on CASSANDRA-4347:
---------------------------------------------
I can reproduce this. The problem seems to be that the new node knows to evict the node it has replaced, but while the rest of the cluster recognizes the IP change, it fails the fat client expiration check and never removes the old IP from gossip. Eventually the new node's quarantine period expires, and it sees the old node again via gossip, causing the looping (but harmless) messages.
I suspect there is another tricky problem with the pernicious hasToken that we removed in CASSANDRA-3747. This won't reproduce in 1.1, and without an imminent (or likely) 1.0.11 release, I'm hesitant to risk breaking anything else here while there is a workaround available.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4347:
----------------------------------------
Comment: was deleted
(was:
I am on vacation till 8/17. For ERP DBA related issues contact Neeren at 650 279-6155. For escalations contact Kevin Railsback.
Thanks,
Harpreet Kaur
)
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: 4347.txt, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt, LocationInfo-hd-279-Data.db
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395983#comment-13395983 ]
Brandon Williams commented on CASSANDRA-4347:
---------------------------------------------
Can you attach the system table from a moved node and also the output from nodetool gossipinfo?
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Priority: Minor
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-4347.
-----------------------------------------
Resolution: Won't Fix
Closing since there isn't going to be a 1.0.13 release and this doesn't exist in 1.1+
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: 4347.txt, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt, LocationInfo-hd-279-Data.db
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4347:
----------------------------------------
Attachment: 4347.txt
bq. Some people on IRC thought so, maybe.
Don't trust them! Especially that driftx joker.
Here's a patch to stop playing the silly game of 'remove-and-ignore this endpoint, then rely on fat client expiration to really get rid of it' and instead calls replacedEndpoint() to remove it directly, as if replace_token had completed for it.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: 4347.txt, LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401441#comment-13401441 ]
Brandon Williams commented on CASSANDRA-4347:
---------------------------------------------
Do you have logs from cass03 around the time of the move? I ask because ALL nodes should have logged the "Nodes [new] and [old] have the same token xxxx. [new] is the new owner" message once, as well as the fat client removal message. If the messages are repeating though, that's a sign that something is wrong.
I ask for cass03 because it still shows 10.12.8.97 as the owner of the token for some reason, which would keep trying to propagate it to the rest of the cluster and cause the messages.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396253#comment-13396253 ]
Brandon Williams commented on CASSANDRA-4347:
---------------------------------------------
LocationInfo is enough
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: dev-cass-post-assassinate-gossipinfo.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4347) IP change of node requires
assassinate to really remove old IP
Posted by "Karl Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Mueller updated CASSANDRA-4347:
------------------------------------
Attachment: kaos-cass03-gossipinfo-postmove.txt
kaos-cass00-gossipinfo-postmove.txt
This is the gossipinfo from two points of view. both postmove.txt files are after the node has changed IPs.
kaos-cass00 is the node which moved IPs. The old IP is 10.12.8.97. The new IP is 10.93.12.10.
kaos-cass03 is a node which did not move. It's IP, if needed, is 10.12.8.87
I also have the gossipinfo from after the assassinate if needed.
> IP change of node requires assassinate to really remove old IP
> --------------------------------------------------------------
>
> Key: CASSANDRA-4347
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4347
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: RHEL6, 64bit
> Reporter: Karl Mueller
> Assignee: Brandon Williams
> Priority: Minor
> Attachments: LocationInfo-hd-279-Data.db, dev-cass-post-assassinate-gossipinfo.txt, kaos-cass00-gossipinfo-postmove.txt, kaos-cass03-gossipinfo-postmove.txt
>
>
> In changing the IP addresses of nodes one-by-one, the node successfully moves itself and its token. Everything works properly.
> However, the node which had its IP changed (but NOT other nodes in the ring) continues to have some type of state associated with the old IP and produces log messages like this:
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:25:01,490 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:25:01,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:25:11,373 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:25:32,380 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:26:32,490 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 Gossiper.java (line 804) InetAddress /10.12.9.157 is now UP
> INFO [GossipStage:1] 2012-06-15 15:26:32,491 StorageService.java (line 1017) Nodes /10.12.9.157 and dev-cass01.sv.walmartlabs.com/10.93.15.11 have the same token 113427455640312821154458202477256070484. Ignoring /10.12.9.157
> INFO [GossipTasks:1] 2012-06-15 15:26:42,402 Gossiper.java (line 818) InetAddress /10.12.9.157 is now dead.
> INFO [GossipTasks:1] 2012-06-15 15:27:03,410 Gossiper.java (line 632) FatClient /10.12.9.157 has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-06-15 15:28:04,533 Gossiper.java (line 838) Node /10.12.9.157 is now part of the cluster
> Other nodes do NOT have the old IP showing up in logs. It's only the node that moved.
> The old IP doesn't show up in ring anywhere or in any other fashion. The cluster seems to be fully operational, so I think it's just a cleanup issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira