You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by mcasandra <mo...@gmail.com> on 2011/02/18 21:35:33 UTC
Error when bringing up 3rd node
I see following error. Is it because I have initial token defined? What token
should I use as initial token?
INFO 12:31:36,689 Finished hinted handoff of 0 rows to endpoint
/172.16.208.12
INFO 12:32:58,448 Joining: getting bootstrap token
ERROR 12:32:58,451 Fatal error: Bootstraping to existing token 0 is not
allowed (decommission/removetoken the old node first).
Bad configuration; unable to start server
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041409.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by Jonathan Ellis <jb...@gmail.com>.
Also, ^ means xor in Java, not exponentiation.
Just use the Python Eric linked. :)
On Fri, Feb 18, 2011 at 3:24 PM, Ching-Cheng Chen
<cc...@evidentsoftware.com> wrote:
> 41
> 82
> 123
> These certainly not correct. Can't just use 2 ^ 127, will overflow
> You can't use Java's primitive type to do this calculation. long only use
> 64 bit.
> You'd need to use BigInteger class to do this calculation.
> Regards,
> Chen
> www.evidentsoftware.com
>
> On Fri, Feb 18, 2011 at 4:04 PM, mcasandra <mo...@gmail.com> wrote:
>>
>> Thanks! This is what I got. Is this right?
>>
>> public class TokenCalc{
>> public static void main(String ...args){
>> int nodes=3;
>> for(int i = 1 ; i <= nodes; i++) {
>> System.out.println( (2 ^ 127) / nodes * i);
>> }
>> }
>> }
>>
>> 41
>> 82
>> 123
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041471.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
>> Nabble.com.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Error when bringing up 3rd node
Posted by Ching-Cheng Chen <cc...@evidentsoftware.com>.
41
82
123
These certainly not correct. Can't just use 2 ^ 127, will overflow
You can't use Java's primitive type to do this calculation. long only use
64 bit.
You'd need to use BigInteger class to do this calculation.
Regards,
Chen
www.evidentsoftware.com
On Fri, Feb 18, 2011 at 4:04 PM, mcasandra <mo...@gmail.com> wrote:
>
> Thanks! This is what I got. Is this right?
>
> public class TokenCalc{
> public static void main(String ...args){
> int nodes=3;
> for(int i = 1 ; i <= nodes; i++) {
> System.out.println( (2 ^ 127) / nodes * i);
> }
> }
> }
>
> 41
> 82
> 123
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041471.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>
Re: Error when bringing up 3rd node
Posted by mcasandra <mo...@gmail.com>.
Could someone please help me understand why this might have happened?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6054768.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by mcasandra <mo...@gmail.com>.
mcasandra wrote:
>
> I am still getting the following:
>
> On node 1:
> ERROR 16:57:31,365 Fatal error: Bootstraping to existing token 0 is not
> allowed (decommission/removetoken the old node first).
>
> On node 2:
> ERROR 16:57:42,300 Fatal error: Bootstraping to existing token
> 56713727820156410577229101238628035242 is not allowed
> (decommission/removetoken the old node first).
> Bad configuration; unable to start server
>
> node 3: is up
>
> node 1 has initial_token: 0 . I think earlier it wasn't set.
>
> node 2 initial_token: 56713727820156410577229101238628035242. Earlier it
> had 0
>
> node 3 initial_token: 113427455640312821154458202477256070484
>
> What should I do now?
>
>
>
>
I am seeing very strange behaviour and makes me really uncomfortable.
Sometimes when I start the nodes I seee:
node 1:
ERROR 17:44:10,718 Error in ThreadPoolExecutor
java.lang.IllegalStateException: replication factor (2) exceeds number of
endpoints (1)
at
org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:188)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:215)
at
org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:792)
at
org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:773)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:587)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:548)
at
org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:720)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:685)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:639)
at
org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:62)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 17:44:10,720 Fatal exception in thread Thread[GossipStage:1,5,main]
java.lang.IllegalStateException: replication factor (2) exceeds number of
endpoints (1)
at
org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:188)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:215)
at
org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:792)
at
org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:773)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:587)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:548)
at
org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:720)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:685)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:639)
at
org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:62)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
node 2:
ERROR 17:43:57,429 Fatal exception in thread Thread[MiscStage:1,5,main]
java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:210)
at
org.apache.cassandra.service.StorageService.getPrimaryRangeForEndpoint(StorageService.java:1289)
at
org.apache.cassandra.service.StorageService.getLocalPrimaryRange(StorageService.java:160)
at
org.apache.cassandra.service.StorageService.getBootstrapToken(StorageService.java:1443)
at
org.apache.cassandra.dht.BootStrapper$BootstrapTokenVerbHandler.doVerb(BootStrapper.java:256)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO 17:44:10,505 Joining: getting bootstrap token
--
If I stop cassandra and start again the error messages disappear. But if I
start the nodes again I sometimes get this error again. In between all this
no data is changing.
I am unable to understand this behaviour and what I need to do if these
things happen in production.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6050903.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by mcasandra <mo...@gmail.com>.
I am still getting the following:
On node 1:
ERROR 16:57:31,365 Fatal error: Bootstraping to existing token 0 is not
allowed (decommission/removetoken the old node first).
On node 2:
ERROR 16:57:42,300 Fatal error: Bootstraping to existing token
56713727820156410577229101238628035242 is not allowed
(decommission/removetoken the old node first).
Bad configuration; unable to start server
node 3: is up
node 1 has initial_token: 0 . I think earlier it wasn't set.
node 2 initial_token: 56713727820156410577229101238628035242. Earlier it had
0
node 3 initial_token: 113427455640312821154458202477256070484
What should I do now?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6050810.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by Ching-Cheng Chen <cc...@evidentsoftware.com>.
If you know you will have 3 nodes, you should set the initial token inside
the cassandra.yaml for each node.
Then you won't need to run nodetool move.
Regards,
Chen
www.evidentsoftware.com
On Fri, Feb 18, 2011 at 5:24 PM, mcasandra <mo...@gmail.com> wrote:
>
> Thanks! I feel so horrible after realizing what mistaked I made :)
>
> After I bring up the new node I just need to run the following on old
> nodes?
>
> 1) New node set the initial token to 56713727820156410577229101238628035242
> 2) start new node
> 3) On second node run nodetool move 113427455640312821154458202477256070484
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041649.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>
Re: Error when bringing up 3rd node
Posted by mcasandra <mo...@gmail.com>.
Thanks! I feel so horrible after realizing what mistaked I made :)
After I bring up the new node I just need to run the following on old nodes?
1) New node set the initial token to 56713727820156410577229101238628035242
2) start new node
3) On second node run nodetool move 113427455640312821154458202477256070484
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041649.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by Ching-Cheng Chen <cc...@evidentsoftware.com>.
try this
BigInteger bi = new BigInteger("2");
BigInteger or = new BigInteger("2");
for (int i=1;i<127;i++) {
or = or.multiply(bi);
}
or = or.divide(new BigInteger("3"));
for (int i=0;i<3;i++) {
System.out.println(or.multiply(new BigInteger(""+i)));
}
which generate
0
56713727820156410577229101238628035242
113427455640312821154458202477256070484
Regards,
Chen
www.evidentsoftware.com
On Fri, Feb 18, 2011 at 4:24 PM, Eric Gilmore <er...@datastax.com> wrote:
> I'm not sure I can say exactly why, but I'm sure those numbers can't be
> correct. One node should be zero and the other values should be very long
> numbers like 85070591730234615865843651857942052863.
>
> We need another Java expert's opinion here, but it looks like your snippet
> may have "integer overflow<http://www.mkyong.com/java/javas-silent-killer-buffer-overflow-careful/>"
> or "integer overload" going on.
>
>
> On Fri, Feb 18, 2011 at 1:04 PM, mcasandra <mo...@gmail.com> wrote:
>
>>
>> Thanks! This is what I got. Is this right?
>>
>> public class TokenCalc{
>> public static void main(String ...args){
>> int nodes=3;
>> for(int i = 1 ; i <= nodes; i++) {
>> System.out.println( (2 ^ 127) / nodes * i);
>> }
>> }
>> }
>>
>> 41
>> 82
>> 123
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041471.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
>> Nabble.com.
>>
>
>
Re: Error when bringing up 3rd node
Posted by Eric Gilmore <er...@datastax.com>.
I'm not sure I can say exactly why, but I'm sure those numbers can't be
correct. One node should be zero and the other values should be very long
numbers like 85070591730234615865843651857942052863.
We need another Java expert's opinion here, but it looks like your snippet
may have "integer
overflow<http://www.mkyong.com/java/javas-silent-killer-buffer-overflow-careful/>"
or "integer overload" going on.
On Fri, Feb 18, 2011 at 1:04 PM, mcasandra <mo...@gmail.com> wrote:
>
> Thanks! This is what I got. Is this right?
>
> public class TokenCalc{
> public static void main(String ...args){
> int nodes=3;
> for(int i = 1 ; i <= nodes; i++) {
> System.out.println( (2 ^ 127) / nodes * i);
> }
> }
> }
>
> 41
> 82
> 123
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041471.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>
Re: Error when bringing up 3rd node
Posted by mcasandra <mo...@gmail.com>.
Thanks! This is what I got. Is this right?
public class TokenCalc{
public static void main(String ...args){
int nodes=3;
for(int i = 1 ; i <= nodes; i++) {
System.out.println( (2 ^ 127) / nodes * i);
}
}
}
41
82
123
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041471.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by Eric Gilmore <er...@datastax.com>.
A Java program should work fine. The Wiki and the DataStax documentation
use a python program for the same purpose:
http://www.datastax.com/docs/0.7/operations/clustering#calculating-tokens
On Fri, Feb 18, 2011 at 12:45 PM, mcasandra <mo...@gmail.com> wrote:
>
> Yes I had set the first node to token 0. I think I read somewhere in the
> docs. What should I do. Should I write a java program to calculate the hash
> for 3 nodes and distribute it accross 3 nodes?
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041430.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>
Re: Error when bringing up 3rd node
Posted by mcasandra <mo...@gmail.com>.
Yes I had set the first node to token 0. I think I read somewhere in the
docs. What should I do. Should I write a java program to calculate the hash
for 3 nodes and distribute it accross 3 nodes?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041430.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Error when bringing up 3rd node
Posted by Eric Gilmore <er...@datastax.com>.
It sounds like one of your existing nodes already has the initial token
zero. Did you set the intial token of the first node you brought online to
zero?
On Fri, Feb 18, 2011 at 12:35 PM, mcasandra <mo...@gmail.com> wrote:
>
> I see following error. Is it because I have initial token defined? What
> token
> should I use as initial token?
>
> INFO 12:31:36,689 Finished hinted handoff of 0 rows to endpoint
> /172.16.208.12
> INFO 12:32:58,448 Joining: getting bootstrap token
> ERROR 12:32:58,451 Fatal error: Bootstraping to existing token 0 is not
> allowed (decommission/removetoken the old node first).
> Bad configuration; unable to start server
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-when-bringing-up-3rd-node-tp6041409p6041409.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>