You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ben Frank <be...@airlust.com> on 2012/09/13 03:15:49 UTC
Schema consistently not propagating to a node.
Hey all,
I'm setting up a new ring on 11 machines with cassandra 1.1.5. All seems
to install fine and startup ok (from tarball) but I'm having an issue when
updating the schema. There's one node that just doesn't want to receive the
schema change. I've tried blowing away my /var/lib/cassandra/* directories
and trying it a few times with the same results. I also tried the
instructions here: http://wiki.apache.org/cassandra/FAQ#schema_disagreement
If I update the schema with cassandra-cli I get the following error:
Waiting for schema agreement...
The schema has not settled in 10 seconds; further migrations are
ill-advised until it does.
Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103,
12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129,
12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133],
b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109]
then if I do a 'describe cluster' I get:
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103,
12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105,
12.19.102.135, 12.19.103.105, 12.19.102.133]
a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109]
if I keep running describe cluster, both the schema UUID's(?) above keep
changing every few seconds. Looking at the logs shows the nodes
are continually doing a flush and compact.
if I start over by blowing away /var/lib/cassandra, and then try the schema
update on the machine which is having trouble receiving it everything works
ok and all nodes are in sync.
Anyone know what's going on here, or what I should look at to troubleshoot
this?
-Ben
Re: Schema consistently not propagating to a node.
Posted by aaron morton <aa...@thelastpickle.com>.
Out of interest, how out of sync where they ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 14/09/2012, at 6:53 AM, Ben Frank <be...@airlust.com> wrote:
> Hi Sergey,
> That was exactly it, thank you!
>
> -Ben
>
> On Thu, Sep 13, 2012 at 12:07 AM, Sergey Tryuber <st...@gmail.com> wrote:
> Hi Ben,
>
> We've seen similar behavior once a time. The cause was unsynchronized time on the nodes. So I can recommend to check that system time is equal over the nodes (especially on that one "bad" server).
>
>
> On 13 September 2012 05:15, Ben Frank <be...@airlust.com> wrote:
> Hey all,
> I'm setting up a new ring on 11 machines with cassandra 1.1.5. All seems to install fine and startup ok (from tarball) but I'm having an issue when updating the schema. There's one node that just doesn't want to receive the schema change. I've tried blowing away my /var/lib/cassandra/* directories and trying it a few times with the same results. I also tried the instructions here: http://wiki.apache.org/cassandra/FAQ#schema_disagreement
>
> If I update the schema with cassandra-cli I get the following error:
>
> Waiting for schema agreement...
> The schema has not settled in 10 seconds; further migrations are ill-advised until it does.
> Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103, 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133], b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109]
>
> then if I do a 'describe cluster' I get:
>
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, 12.19.102.135, 12.19.103.105, 12.19.102.133]
>
> a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109]
>
> if I keep running describe cluster, both the schema UUID's(?) above keep changing every few seconds. Looking at the logs shows the nodes are continually doing a flush and compact.
>
> if I start over by blowing away /var/lib/cassandra, and then try the schema update on the machine which is having trouble receiving it everything works ok and all nodes are in sync.
>
> Anyone know what's going on here, or what I should look at to troubleshoot this?
>
> -Ben
>
>
Re: Schema consistently not propagating to a node.
Posted by Ben Frank <be...@airlust.com>.
Hi Sergey,
That was exactly it, thank you!
-Ben
On Thu, Sep 13, 2012 at 12:07 AM, Sergey Tryuber <st...@gmail.com> wrote:
> Hi Ben,
>
> We've seen similar behavior once a time. The cause was unsynchronized time
> on the nodes. So I can recommend to check that system time is equal over
> the nodes (especially on that one "bad" server).
>
>
> On 13 September 2012 05:15, Ben Frank <be...@airlust.com> wrote:
>
>> Hey all,
>> I'm setting up a new ring on 11 machines with cassandra 1.1.5. All
>> seems to install fine and startup ok (from tarball) but I'm having an issue
>> when updating the schema. There's one node that just doesn't want to
>> receive the schema change. I've tried blowing away my /var/lib/cassandra/*
>> directories and trying it a few times with the same results. I also tried
>> the instructions here:
>> http://wiki.apache.org/cassandra/FAQ#schema_disagreement
>>
>> If I update the schema with cassandra-cli I get the following error:
>>
>> Waiting for schema agreement...
>> The schema has not settled in 10 seconds; further migrations are
>> ill-advised until it does.
>> Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103,
>> 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129,
>> 12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133],
>> b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109]
>>
>> then if I do a 'describe cluster' I get:
>>
>> Cluster Information:
>> Snitch: org.apache.cassandra.locator.SimpleSnitch
>> Partitioner: org.apache.cassandra.dht.RandomPartitioner
>> Schema versions:
>> 3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103,
>> 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105,
>> 12.19.102.135, 12.19.103.105, 12.19.102.133]
>>
>> a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109]
>>
>> if I keep running describe cluster, both the schema UUID's(?) above keep
>> changing every few seconds. Looking at the logs shows the nodes
>> are continually doing a flush and compact.
>>
>> if I start over by blowing away /var/lib/cassandra, and then try the
>> schema update on the machine which is having trouble receiving
>> it everything works ok and all nodes are in sync.
>>
>> Anyone know what's going on here, or what I should look at to
>> troubleshoot this?
>>
>> -Ben
>>
>
>
Re: Schema consistently not propagating to a node.
Posted by Sergey Tryuber <st...@gmail.com>.
Hi Ben,
We've seen similar behavior once a time. The cause was unsynchronized time
on the nodes. So I can recommend to check that system time is equal over
the nodes (especially on that one "bad" server).
On 13 September 2012 05:15, Ben Frank <be...@airlust.com> wrote:
> Hey all,
> I'm setting up a new ring on 11 machines with cassandra 1.1.5. All
> seems to install fine and startup ok (from tarball) but I'm having an issue
> when updating the schema. There's one node that just doesn't want to
> receive the schema change. I've tried blowing away my /var/lib/cassandra/*
> directories and trying it a few times with the same results. I also tried
> the instructions here:
> http://wiki.apache.org/cassandra/FAQ#schema_disagreement
>
> If I update the schema with cassandra-cli I get the following error:
>
> Waiting for schema agreement...
> The schema has not settled in 10 seconds; further migrations are
> ill-advised until it does.
> Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103,
> 12.19.110.103, 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129,
> 12.19.110.105, 172.19.102.135, 12.19.103.105, 12.19.102.133],
> b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109]
>
> then if I do a 'describe cluster' I get:
>
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103,
> 12.19.102.137, 12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105,
> 12.19.102.135, 12.19.103.105, 12.19.102.133]
>
> a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109]
>
> if I keep running describe cluster, both the schema UUID's(?) above keep
> changing every few seconds. Looking at the logs shows the nodes
> are continually doing a flush and compact.
>
> if I start over by blowing away /var/lib/cassandra, and then try the
> schema update on the machine which is having trouble receiving
> it everything works ok and all nodes are in sync.
>
> Anyone know what's going on here, or what I should look at to troubleshoot
> this?
>
> -Ben
>