You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/04/04 15:21:05 UTC
[jira] [Updated] (CASSANDRA-2415) Ec2Snitch changing tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2415:
--------------------------------------
Component/s: Core
Priority: Minor (was: Major)
Fix Version/s: 0.7.5
Assignee: Brandon Williams
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira