You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Patrik Modesto <pa...@gmail.com> on 2011/03/08 16:23:02 UTC
problem with bootstrap
Hi,
I've small test cluster, 2 servers, both running successfully
cassandra 0.7.3. I've three keyspaces, two with RF1, one with RF3. Now
when I try to bootstrap 3rd server (empty initial_token,
auto_bootstrap: true), I get this exception on the new server.
INFO 23:13:43,229 Joining: getting bootstrap token
INFO 23:13:43,258 New token will be
127097301048222781806986236020167142093 to assume load from
/10.0.18.99
INFO 23:13:43,259 switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/mnt/disk8/cassandra/data/CommitLog-1299622332896.log',
position=1578072)
INFO 23:13:43,259 Enqueuing flush of
Memtable-LocationInfo@1526249359(106 bytes, 3 operations)
INFO 23:13:43,259 Writing Memtable-LocationInfo@1526249359(106 bytes,
3 operations)
INFO 23:13:43,276 Completed flushing
/mnt/disk3/cassandra/data/system/LocationInfo-f-2-Data.db (211 bytes)
INFO 23:13:43,277 Joining: sleeping 30000 ms for pending range setup
INFO 23:14:13,277 Bootstrapping
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
Caused by: java.lang.IllegalStateException: replication factor (3)
exceeds number of endpoints (2)
at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:212)
at org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:198)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:403)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194)
at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:217)
... 5 more
Cannot load daemon
Service exit with a return value of 3
On the other servers I get:
ERROR 15:54:24,670 Error in ThreadPoolExecutor
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)
at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
at org.apache.cassandra.service.StorageService.handleStateLeaving(StorageService.java:797)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:651)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:763)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:753)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:670)
at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 15:54:24,672 Fatal exception in thread Thread[GossipStage:1,5,main]
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)
at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
at org.apache.cassandra.service.StorageService.handleStateLeaving(StorageService.java:797)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:651)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:763)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:753)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:670)
at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Removing the keyspace with RF3 fixed the problem and boostrap went
well, but why is there a problem with less nodes than servers? I can
imagine a situation when I would need to remove nodes from cluster and
get to the point of having less servers than is the maximum RF used.
I'd then be unable to bootstrap the new servers to the cluster.
Removing the keyspace is not an option in production environment.
Thanks,
Patrik
Re: problem with bootstrap
Posted by Peter Schuller <pe...@infidyne.com>.
> Bootstrapping uses the same mechanisms as a repair to streams data from other nodes. This can be a heavy weight process and you may want to control when it starts.
>
> Joining the ring just tells the other nodes you exists and this is your token.
And in general, except when initially setting up a cluster, you would
not normally want to have a node join the ring without auto_bootstrap
enabled since it will serve requests yet be void of any data.
--
/ Peter Schuller
Re: problem with bootstrap
Posted by aaron morton <aa...@thelastpickle.com>.
Bootstrapping uses the same mechanisms as a repair to streams data from other nodes. This can be a heavy weight process and you may want to control when it starts.
Joining the ring just tells the other nodes you exists and this is your token.
A
On 10/03/2011, at 9:27 AM, mcasandra wrote:
> Thanks!
>
> aaron morton wrote:
>>
>>
>> The issue I think you and Patrik are seeing occurs when you *remove* nodes
>> from the ring. The ring does not know if they are up or down. E.g. you
>> have a ring of 3 nodes, and add a keyspace with RF 3. Then for whatever
>> reason 2 nodes are removed from the ring. When bootstrapping a node into
>> this ring it will fail because it detects the cluster does not have enough
>> *endpoints* (different to up nodes) to support the keyspace.
>>
>>
> What causes a node to remove? All I did was kill -9 and then sudo cassandra
> to start the node.
>
>
>
>> IMHO bootstrapping is the process of pulling data the *new* node is
>> responsible for from other nodes in the ring. This is different to joining
>> the ring.
>>
>
> How is this different than joining the ring? It will be good to see some
> example and the difference.
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6155334.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by mcasandra <mo...@gmail.com>.
mcasandra wrote:
>
>
> aaron morton wrote:
>>
>>
>> The issue I think you and Patrik are seeing occurs when you *remove*
>> nodes from the ring. The ring does not know if they are up or down. E.g.
>> you have a ring of 3 nodes, and add a keyspace with RF 3. Then for
>> whatever reason 2 nodes are removed from the ring. When bootstrapping a
>> node into this ring it will fail because it detects the cluster does not
>> have enough *endpoints* (different to up nodes) to support the keyspace.
>>
>>
> Thanks for more info. However I am still not understanding why I am
> running in this situation since this node was once up like other node. In
> your previous post you mentioned that the node got removed. I am trying to
> understand what that really means and what causes a node to remove? All I
> did was kill -9 and then sudo cassandra to start the node.
>
I am still trying to see how to find the root cause of this behaviour. I
wonder if this were to happen in production how will we debug or what will
we do :(
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6158996.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by mcasandra <mo...@gmail.com>.
aaron morton wrote:
>
>
> The issue I think you and Patrik are seeing occurs when you *remove* nodes
> from the ring. The ring does not know if they are up or down. E.g. you
> have a ring of 3 nodes, and add a keyspace with RF 3. Then for whatever
> reason 2 nodes are removed from the ring. When bootstrapping a node into
> this ring it will fail because it detects the cluster does not have enough
> *endpoints* (different to up nodes) to support the keyspace.
>
>
In your previous post you mentioned that the node got removed. I am trying
to understand what that really means and what causes a node to remove? All I
did was kill -9 and then sudo cassandra to start the node.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6156250.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by mcasandra <mo...@gmail.com>.
Thanks!
aaron morton wrote:
>
>
> The issue I think you and Patrik are seeing occurs when you *remove* nodes
> from the ring. The ring does not know if they are up or down. E.g. you
> have a ring of 3 nodes, and add a keyspace with RF 3. Then for whatever
> reason 2 nodes are removed from the ring. When bootstrapping a node into
> this ring it will fail because it detects the cluster does not have enough
> *endpoints* (different to up nodes) to support the keyspace.
>
>
What causes a node to remove? All I did was kill -9 and then sudo cassandra
to start the node.
> IMHO bootstrapping is the process of pulling data the *new* node is
> responsible for from other nodes in the ring. This is different to joining
> the ring.
>
How is this different than joining the ring? It will be good to see some
example and the difference.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6155334.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by aaron morton <aa...@thelastpickle.com>.
The definition of "down" is important here.
Down refers to a node that has joined the ring, so the other nodes know of it's existence and the range it is storing, which is not responding to gossip messages. While it is down it is still considered an endpoint. The error you and Patrik saw refers to the number of endpoints in the ring, not the number of Up nodes. When doing dev I have a 2 nodes cluster on my laptop with rf=2, it's fine to bring the nodes in the cluster up one at a time.
The issue I think you and Patrik are seeing occurs when you *remove* nodes from the ring. The ring does not know if they are up or down. E.g. you have a ring of 3 nodes, and add a keyspace with RF 3. Then for whatever reason 2 nodes are removed from the ring. When bootstrapping a node into this ring it will fail because it detects the cluster does not have enough *endpoints* (different to up nodes) to support the keyspace.
One thing I want to double check is that the node doing the bootstrap considers it's self when calculating the number of end points. Some of the things you and Patrik said about bootstrapping node 3 into a ring of 3 with rf=3 made me want to check.
IMHO bootstrapping is the process of pulling data the *new* node is responsible for from other nodes in the ring. This is different to joining the ring.
Hope that helps.
Aaron
On 9/03/2011, at 10:54 AM, mcasandra wrote:
> I think this not the right functionality and it is really odd that you can't
> successfully bring it online without turning off bootstrap BUT you can bring
> it online by turning auto_boostrap off and then run nodetool repair
> afterwards.
>
> Also, if that's the case then when one node goes down, say out of 3 one node
> goes down then should cassandra eject other nodes as well?? Why should
> cassandra exit on startup? That node could at least serve other keyspaces
> and alleviate load while returning errors to the client for those keyspaces
> where RF cannot be met.
>
> As noted in my other post regarding similar issue that I reported, I have
> also seen wierd behaviour where I had 2 nodes down out of 3 and I was able
> to bring up one of the nodes except the remaining one. You would think that
> no nodes will come up but I really think there is a problem here.
>
>
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6145100.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by mcasandra <mo...@gmail.com>.
I think this not the right functionality and it is really odd that you can't
successfully bring it online without turning off bootstrap BUT you can bring
it online by turning auto_boostrap off and then run nodetool repair
afterwards.
Also, if that's the case then when one node goes down, say out of 3 one node
goes down then should cassandra eject other nodes as well?? Why should
cassandra exit on startup? That node could at least serve other keyspaces
and alleviate load while returning errors to the client for those keyspaces
where RF cannot be met.
As noted in my other post regarding similar issue that I reported, I have
also seen wierd behaviour where I had 2 nodes down out of 3 and I was able
to bring up one of the nodes except the remaining one. You would think that
no nodes will come up but I really think there is a problem here.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6145100.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by aaron morton <aa...@thelastpickle.com>.
Thanks, will try to look into it.
Aaron
On 14 Mar 2011, at 20:43, Patrik Modesto wrote:
> On Fri, Mar 11, 2011 at 22:31, Aaron Morton <aa...@thelastpickle.com> wrote:
>> The assertion is interesting. Can you reproduce it with logging at debug and post the results? Could you try to reproduce it with a clean cluster?
>
> It was on a clean cluster last time. Anyway I started clean cluster
> again, repeated the same steps with DEBUG enabled. This is the part of
> log when the AssertionError happend:
>
> DEBUG 08:32:50,513 reading RangeSliceCommand{keyspace='Context',
> column_family='Url', super_column=null,
> predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00
> 00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 07
> 0C 00 01 0B 00 03 00 00 00 03 55 72 6C 00 0C 00 02 0C 00 02 0B 00 01
> 00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65
> 5F 73 6C 69 63 65 73 00 00 00 07 0C 00 01 0B 00 03 00 00 00 03 55 72
> 6C 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00,
> reversed:false, count:2147483647)),
> range=(48888839275280018186837186594882659755,53884050462475230985987955197486389848],
> max_keys=100} from /10.0.18.99
> DEBUG 08:32:50,513 reading RangeSliceCommand{keyspace='Context',
> column_family='Url', super_column=null,
> predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00
> 00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 07
> 0C 00 01 0B 00 03 00 00 00 03 55 72 6C 00 0C 00 02 0C 00 02 0B 00 01
> 00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65
> 5F 73 6C 69 63 65 73 00 00 00 07 0C 00 01 0B 00 03 00 00 00 03 55 72
> 6C 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00,
> reversed:false, count:2147483647)),
> range=(48888839275280018186837186594882659755,53884050462475230985987955197486389848],
> max_keys=100} from /10.0.18.129
> DEBUG 08:32:50,515 Processing response on a callback from 159@/10.0.18.99
> DEBUG 08:32:50,515 Processing response on a callback from 160@/10.0.18.129
> ERROR 08:32:50,518 Fatal exception in thread
> Thread[RequestResponseStage:3,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> It happens on every node I run the request on, the others report
> nothing on that particular request.
>
> HTH,
> P.
Re: problem with bootstrap
Posted by Patrik Modesto <pa...@gmail.com>.
On Fri, Mar 11, 2011 at 22:31, Aaron Morton <aa...@thelastpickle.com> wrote:
> The assertion is interesting. Can you reproduce it with logging at debug and post the results? Could you try to reproduce it with a clean cluster?
It was on a clean cluster last time. Anyway I started clean cluster
again, repeated the same steps with DEBUG enabled. This is the part of
log when the AssertionError happend:
DEBUG 08:32:50,513 reading RangeSliceCommand{keyspace='Context',
column_family='Url', super_column=null,
predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00
00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 07
0C 00 01 0B 00 03 00 00 00 03 55 72 6C 00 0C 00 02 0C 00 02 0B 00 01
00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65
5F 73 6C 69 63 65 73 00 00 00 07 0C 00 01 0B 00 03 00 00 00 03 55 72
6C 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00,
reversed:false, count:2147483647)),
range=(48888839275280018186837186594882659755,53884050462475230985987955197486389848],
max_keys=100} from /10.0.18.99
DEBUG 08:32:50,513 reading RangeSliceCommand{keyspace='Context',
column_family='Url', super_column=null,
predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00
00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 07
0C 00 01 0B 00 03 00 00 00 03 55 72 6C 00 0C 00 02 0C 00 02 0B 00 01
00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65
5F 73 6C 69 63 65 73 00 00 00 07 0C 00 01 0B 00 03 00 00 00 03 55 72
6C 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00,
reversed:false, count:2147483647)),
range=(48888839275280018186837186594882659755,53884050462475230985987955197486389848],
max_keys=100} from /10.0.18.129
DEBUG 08:32:50,515 Processing response on a callback from 159@/10.0.18.99
DEBUG 08:32:50,515 Processing response on a callback from 160@/10.0.18.129
ERROR 08:32:50,518 Fatal exception in thread
Thread[RequestResponseStage:3,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
It happens on every node I run the request on, the others report
nothing on that particular request.
HTH,
P.
Re: problem with bootstrap
Posted by Aaron Morton <aa...@thelastpickle.com>.
IMHO creating a keyspace with RF higher than the number of nodes sounds like a bug. It puts the cluster into a bad place. It may even be a regression, will take a look at the code.
The assertion is interesting. Can you reproduce it with logging at debug and post the results? Could you try to reproduce it with a clean cluster?
Thanks
Aaron
On 11/03/2011, at 10:24 PM, Patrik Modesto <pa...@gmail.com> wrote:
> Unfortunately I can't provide the info, I deleted it. It was in wery
> strange state.
>
> I started with new cluster today, 2 nodes, each with
> auto_bootstrap:true. I can create a keyspace with RF=3, but I can't
> insert any data in it. It didn't happen with the old cluster which
> made me think. How could I insert data in the old cluster in keyspace
> with RF=3 but with just 2 nodes? I found out that the cluster had 3
> nodes for short time in the past. We had to remove/return one node but
> that was enough for the cluster to accept writes to keyspace with RF=3
> even with just 2 nodes.
>
> So I tried to recreate the cluster state:
>
> I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true
>
> 1) setup & run node1 - success
>
> 2) create keyspace Context with rf=3" and create CF Url via
> cassandra-cli - success
>
> 3) list Url - Internal error processing get_range_slicesl
> node1:
> ERROR 09:46:28,725 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (1)
>
> 4) setup & run node2 - success
>
> 5) list Url on node1 - Internal error processing get_range_slicesl
> node1:
> ERROR 09:46:28,725 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (1)
>
> 6) list Url on node2 - Internal error processing get_range_slicesl
> node2:
> ERROR 09:50:54,231 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
>
> 7) insert on node1 - Internal error processing insert
> node1:
> ERROR 09:53:11,669 Internal error processing insert
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
>
> 8) insert on node2 - Internal error processing insert
> node2:
> ERROR 09:53:54,833 Internal error processing insert
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
>
> 9) setup & run node3 - success
>
> 10) list Url on node1 - success
>
> 11) insert in Url on node1 - success
>
> 12) stop cassandra on node3 - success
>
> 13) list & insert on node1&2 - success
>
> 14) loadbalance on node1 - Exception in thread "main"
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
>
> 15) setup & run node4 - success
>
> 16) list Url on node4 - success BUT
> node4:
> ERROR 10:05:38,452 Fatal exception in thread
> Thread[RequestResponseStage:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 10:05:38,462 Fatal exception in thread
> Thread[RequestResponseStage:17,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> 17) loadbalance on node1 - success
>
> 18) list Url on node4 - success BUT
> node4:
> ERROR 10:09:58,251 Fatal exception in thread
> Thread[RequestResponseStage:18,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 10:09:58,257 Fatal exception in thread
> Thread[RequestResponseStage:5,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> 19) repair on node4 - after long long wait I killed it, non of the
> nodes report any error
>
> 20) list Url on node1 - success BUT
> node1:
> ERROR 10:18:53,715 Fatal exception in thread
> Thread[RequestResponseStage:6,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> So, I can't get the cluster to the state it was before the reinstalled
> it, where I couldn't bootstrap new node. I hope it was just
> combination of cassandra upgrades and lots of scheme changes and that
> it won't happen in production. OTOH there is the AssertionError which
> doesn't look good but I can insert/retrieve the data.
>
> Regards,
> Patrik
Re: problem with bootstrap
Posted by Patrik Modesto <pa...@gmail.com>.
Unfortunately I can't provide the info, I deleted it. It was in wery
strange state.
I started with new cluster today, 2 nodes, each with
auto_bootstrap:true. I can create a keyspace with RF=3, but I can't
insert any data in it. It didn't happen with the old cluster which
made me think. How could I insert data in the old cluster in keyspace
with RF=3 but with just 2 nodes? I found out that the cluster had 3
nodes for short time in the past. We had to remove/return one node but
that was enough for the cluster to accept writes to keyspace with RF=3
even with just 2 nodes.
So I tried to recreate the cluster state:
I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true
1) setup & run node1 - success
2) create keyspace Context with rf=3" and create CF Url via
cassandra-cli - success
3) list Url - Internal error processing get_range_slicesl
node1:
ERROR 09:46:28,725 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (1)
4) setup & run node2 - success
5) list Url on node1 - Internal error processing get_range_slicesl
node1:
ERROR 09:46:28,725 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (1)
6) list Url on node2 - Internal error processing get_range_slicesl
node2:
ERROR 09:50:54,231 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)
7) insert on node1 - Internal error processing insert
node1:
ERROR 09:53:11,669 Internal error processing insert
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)
8) insert on node2 - Internal error processing insert
node2:
ERROR 09:53:54,833 Internal error processing insert
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)
9) setup & run node3 - success
10) list Url on node1 - success
11) insert in Url on node1 - success
12) stop cassandra on node3 - success
13) list & insert on node1&2 - success
14) loadbalance on node1 - Exception in thread "main"
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)
15) setup & run node4 - success
16) list Url on node4 - success BUT
node4:
ERROR 10:05:38,452 Fatal exception in thread
Thread[RequestResponseStage:1,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 10:05:38,462 Fatal exception in thread
Thread[RequestResponseStage:17,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
17) loadbalance on node1 - success
18) list Url on node4 - success BUT
node4:
ERROR 10:09:58,251 Fatal exception in thread
Thread[RequestResponseStage:18,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 10:09:58,257 Fatal exception in thread
Thread[RequestResponseStage:5,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
19) repair on node4 - after long long wait I killed it, non of the
nodes report any error
20) list Url on node1 - success BUT
node1:
ERROR 10:18:53,715 Fatal exception in thread
Thread[RequestResponseStage:6,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
So, I can't get the cluster to the state it was before the reinstalled
it, where I couldn't bootstrap new node. I hope it was just
combination of cassandra upgrades and lots of scheme changes and that
it won't happen in production. OTOH there is the AssertionError which
doesn't look good but I can insert/retrieve the data.
Regards,
Patrik
Re: problem with bootstrap
Posted by Peter Schuller <pe...@infidyne.com>.
> Could it be because once auto_bootstrap is off it's off forever?
I am not entirely sure if this answers your question (I revisisted the
thread history but I'm a bit confused myself):
If by that you mean that given a node which was started with
auto_bootstrap=false, and it successfully joined the ring etc, that
changing autobootstrap to true in the configuration and restarting has
no effect - then yes. The auto_bootstrap and initial_token settings
only make sense on a node which has not yet been made part of the
ring.
The only time at which it might bootstrap (word used in the sense of
"populate with data by grabbing data form other nodes") is on initial
start-up. As soon as start-up completes, it is written to the system
tables that an initial start-up has been completed.
(See StorageService.joinTokenRing(); the
SystemTable.setBootstrapped(true) call at the end is what causes a
future start-up to never trigger bootstrapping.)
--
/ Peter Schuller
Re: problem with bootstrap
Posted by mcasandra <mo...@gmail.com>.
I am completely confused. I repeated same test after turning on
auto_bootstrap to true and it worked this time. I did it exactly same way
where I killed 2 nodes and this time it started with no issues.
Could it be because once auto_bootstrap is off it's off forever?
I am using hector and upgraded hector this morning.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/problem-with-bootstrap-tp6127315p6159679.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: problem with bootstrap
Posted by aaron morton <aa...@thelastpickle.com>.
Can you include this info...
- output from nodetool ring for all nodes so we can see whats in the ring
- what you've run on the node you are trying to bring in
- the nodetool command you are trying to run
- error logs
In general asking the cluster to replicate data more times than the number of nodes it has is a bad thing. How did you create the keyspace ?
Aaron
On 11 Mar 2011, at 03:45, Patrik Modesto wrote:
> Hi,
>
> I'm stil fighting the
> Exception in thread "main" java.lang.IllegalStateException:
> replication factor (3) exceeds number of endpoints (2).
>
> When I have a 2-server cluster, create Keyspace with RF 3, I'm able to
> add (without auto_bootstrap) another node but cluster nodetool
> commands don't work and fail with the exception above. The new node
> serve data but I can't do loadbalance, decommission or move on the
> cluster.
>
> Patrik
Re: problem with bootstrap
Posted by Patrik Modesto <pa...@gmail.com>.
Hi,
I'm stil fighting the
Exception in thread "main" java.lang.IllegalStateException:
replication factor (3) exceeds number of endpoints (2).
When I have a 2-server cluster, create Keyspace with RF 3, I'm able to
add (without auto_bootstrap) another node but cluster nodetool
commands don't work and fail with the exception above. The new node
serve data but I can't do loadbalance, decommission or move on the
cluster.
Patrik
Re: problem with bootstrap
Posted by aaron morton <aa...@thelastpickle.com>.
I've seen this around a couple of times now.
On reason to fail if there is not enough nodes to meet the replication factor is that CL.ALL requests cannot be processed. You could make the argument that we can get into that state at any time is a node is down. But this error is their never been enough nodes in the ring regardless of their up/down state. So cassandra will never be able to meet the replication guarantees for the keyspace. E.g. if you kicked off a repair it would not leave the cluster in the expected state.
Not sure if this is the official reason, just my thinking. And their may be other reasons.
Sounds like you've made progress though.
Cheers
Aaron
On 9/03/2011, at 4:23 AM, Patrik Modesto wrote:
> Hi,
>
> I've small test cluster, 2 servers, both running successfully
> cassandra 0.7.3. I've three keyspaces, two with RF1, one with RF3. Now
> when I try to bootstrap 3rd server (empty initial_token,
> auto_bootstrap: true), I get this exception on the new server.
>
> INFO 23:13:43,229 Joining: getting bootstrap token
> INFO 23:13:43,258 New token will be
> 127097301048222781806986236020167142093 to assume load from
> /10.0.18.99
> INFO 23:13:43,259 switching in a fresh Memtable for LocationInfo at
> CommitLogContext(file='/mnt/disk8/cassandra/data/CommitLog-1299622332896.log',
> position=1578072)
> INFO 23:13:43,259 Enqueuing flush of
> Memtable-LocationInfo@1526249359(106 bytes, 3 operations)
> INFO 23:13:43,259 Writing Memtable-LocationInfo@1526249359(106 bytes,
> 3 operations)
> INFO 23:13:43,276 Completed flushing
> /mnt/disk3/cassandra/data/system/LocationInfo-f-2-Data.db (211 bytes)
> INFO 23:13:43,277 Joining: sleeping 30000 ms for pending range setup
> INFO 23:14:13,277 Bootstrapping
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
> Caused by: java.lang.IllegalStateException: replication factor (3)
> exceeds number of endpoints (2)
> at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:212)
> at org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:198)
> at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
> at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525)
> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453)
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:403)
> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194)
> at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:217)
> ... 5 more
> Cannot load daemon
> Service exit with a return value of 3
>
> On the other servers I get:
>
> ERROR 15:54:24,670 Error in ThreadPoolExecutor
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
> at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
> at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
> at org.apache.cassandra.service.StorageService.handleStateLeaving(StorageService.java:797)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:651)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:763)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:753)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:670)
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 15:54:24,672 Fatal exception in thread Thread[GossipStage:1,5,main]
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60)
> at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
> at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
> at org.apache.cassandra.service.StorageService.handleStateLeaving(StorageService.java:797)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:651)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:763)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:753)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:670)
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
>
> Removing the keyspace with RF3 fixed the problem and boostrap went
> well, but why is there a problem with less nodes than servers? I can
> imagine a situation when I would need to remove nodes from cluster and
> get to the point of having less servers than is the maximum RF used.
> I'd then be unable to bootstrap the new servers to the cluster.
> Removing the keyspace is not an option in production environment.
>
> Thanks,
> Patrik