You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sasha Dolgy (JIRA)" <ji...@apache.org> on 2011/04/04 14:15:05 UTC
[jira] [Created] (CASSANDRA-2415) Ec2Snitch changing tokens
Ec2Snitch changing tokens
-------------------------
Key: CASSANDRA-2415
URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.7.4
Environment: Amazon EC2 -- 4 nodes
Reporter: Sasha Dolgy
A new 4 node 0.7.4 cluster on Amazon EC2
1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
2. Brought up a second node, with the first node defined as the seed. No visible issues.
3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
Initially, I -did not- define tokens for the nodes.
When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
The tokens for the 4 nodes:
0
19095547144942516281182777765338228798
56713727820156410577229101238628035242
170141183460469231731687303715884105726
When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
:~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
endpoint /10.0.0.2
INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
to endpoint /10.0.0.2
INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [GossipStage:2] 2011-03-23 00:37:55,382
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[GossipStage:2,5,main]
java.lang.RuntimeException: Bootstrap Token collision between
/10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
:~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
(line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
115) Token 19095547144942516281182777765338228798 changing ownership
from /10.0.0.3 to /10.0.0.2
:~$ nodetool -h 10.0.0.1 -p 9090 ring
Address Status State Load Owns Token
170141183460469231731687303715884105726
10.0.0.1 Up Normal 99.31 KB 0.00% 0
10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
:~$
The EC2 nodes are all part of the same region (Singapore).
I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Sasha Dolgy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015713#comment-13015713 ]
Sasha Dolgy commented on CASSANDRA-2415:
----------------------------------------
a token was not set in the cassandra.yaml initially -- when nodes 3 & 4 were added, the collision errors appear:
cluster_name: 'FOO_Cluster'
initial_token:
auto_bootstrap: true
hinted_handoff_enabled: true
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partitioner: org.apache.cassandra.dht.RandomPartitioner
data_file_directories:
- /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches
commitlog_rotation_threshold_in_mb: 128
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
seeds:
- ec2-xx-xx-xx-xx.ap-southeast-1.compute.amazonaws.com
- ec2-yy-yy-yy-yy.ap-southeast-1.compute.amazonaws.com
disk_access_mode: auto
concurrent_reads: 8
concurrent_writes: 32
sliced_buffer_size_in_kb: 64
storage_port: 7000
rpc_port: 9160
rpc_keepalive: true
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
snapshot_before_compaction: false
binary_memtable_throughput_in_mb: 256
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch
dynamic_snitch: true
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.0
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Sasha Dolgy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015897#comment-13015897 ]
Sasha Dolgy commented on CASSANDRA-2415:
----------------------------------------
No, it does not, or rather, I should say, has not ...
I appreciate it's pretty poor when the bug can't easily be reproduced. I don't have spare capacity at the moment to create another 4 node cluster to try and recreate what could be my mistake, or a legitimate bug.
I'll keep my eye on it
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2415.
---------------------------------------
Resolution: Cannot Reproduce
Fix Version/s: (was: 0.7.5)
Assignee: (was: Brandon Williams)
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Priority: Minor
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2415:
--------------------------------------
Component/s: Core
Priority: Minor (was: Major)
Fix Version/s: 0.7.5
Assignee: Brandon Williams
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Sasha Dolgy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015718#comment-13015718 ]
Sasha Dolgy commented on CASSANDRA-2415:
----------------------------------------
no. they were added one at a time and only once the previous node had fully joined the ring 100% and state in nodetool ring was "Normal".
problem was stopped by changing the endpoint_snitch to SimpleSnitch in the yaml on each node, restarting each node in sequence, waiting for it to resume normal operations (as per the system.log) before continuing with the next node. have not seen the error since moving away from Ec2Snitch
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015633#comment-13015633 ]
Brandon Williams commented on CASSANDRA-2415:
---------------------------------------------
What was the token set to in cassandra.yaml? The snitch can't change the token, so this sounds like a normal bootstrap collision; either the token specified conflicted, or the bootstrap rules for automatic selection weren't adhered to (wait 2 mins between every bootstrap, etc.)
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015715#comment-13015715 ]
Brandon Williams commented on CASSANDRA-2415:
---------------------------------------------
Were they added simultaneously?
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015719#comment-13015719 ]
Brandon Williams commented on CASSANDRA-2415:
---------------------------------------------
If you switch back to ec2snitch does the problem return?
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2415) Ec2Snitch changing tokens
Posted by "Sasha Dolgy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017457#comment-13017457 ]
Sasha Dolgy commented on CASSANDRA-2415:
----------------------------------------
not seen anything since ... so maybe this is attributed to user error and not waiting long enough between bringing new nodes into the ring ..
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml (first one i did it for because of the errors I saw with node 3) ... everything goes well at first in joining the ring, etc.....then I see the following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22% 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78% 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira