You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Langston, Jim" <Ji...@compuware.com> on 2013/07/09 19:08:40 UTC
alter column family ?
Hi all,
I am trying to alter a column family to change gc_grace_seconds, and now,
any of the properties
The sequence:
use ks ;
alter table CF with gc_grace_seconds=864000 ;
When listing the CF, gc_grace_seconds is set to 0, after
running the CLI, gc_grace_seconds is still set to 0.
I tried change the comment property, but this did not
change either.
Using the same keyspace, I created another table
and executed both the gc_grace_seconds change and
the comments change. Both of these successfully changed.
I do not know why I cannot change the value on a table that
was created earlier. I have tried shutting down the other nodes
in the cluster and attempted to change just one of the nodes,
also, I tried to stop/start Cassandra on that node as well to
see if the change would take effect that way.
Is there a permission ?
Where are the property values kept?
Thanks,
Jim
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Jul 17, 2013 at 7:23 PM, Langston, Jim
<Ji...@compuware.com>wrote:
> As a follow up – I did upgrade the cluster to 1.2.6 and that
> did take care of the issue. The upgrade went very smoothly,
> the longest part was being thorough on the configuration
> files, but I was able to able to quickly update the schema's
> after restarting the cluster.
>
Glad to hear it!
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
As a follow up – I did upgrade the cluster to 1.2.6 and that
did take care of the issue. The upgrade went very smoothly,
the longest part was being thorough on the configuration
files, but I was able to able to quickly update the schema's
after restarting the cluster.
Jim
From: "Langston, Jim" <Ji...@compuware.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Thu, 11 Jul 2013 18:25:42 +0000
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
Was just looking at a bug with uppercase , could that be the error ?
And, yes, definitely saved off the original system keyspaces.
I'm tailing the logs when running the cassandra-cli, but I do not
see anything in the logs ..
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Thu, 11 Jul 2013 11:07:55 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Thu, Jul 11, 2013 at 11:00 AM, Langston, Jim <Ji...@compuware.com>> wrote:
I went through the whole sequence again and now have gotten to the point of
being able to try and pull in the schema, but now getting this error from the one
node I'm executing on.
[default@unknown] create keyspace OTracker
9209ec36-3b3f-3e24-9dfb-8a45a5b29a2a
Waiting for schema agreement...
... schemas agree across the cluster
NotFoundException()
This is pretty unusual.
All the nodes see each other and are available, all only contain a system
schema, none have a OTracker schema
If you look in the logs for schema related stuff when you try to create OTracker, what do you see?
Do you see the above UUID schema version in the logs?
At this point I am unable to suggest anything other than upgrading to the head of 1.1 line and try to create your keyspace there. There should be no chance of old state being implicated in your now stuck schema, so it seems likely that the problem has re-occured due to the version of Cassandra you are running.
Sorry I am unable to be of more assistance and that my advice appears to have resulted in your cluster being in worse condition than when you started. I probably mentioned but will do so again that if you have the old system keyspace directories, you can stop cassandra on all nodes and then revert to them.
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
Was just looking at a bug with uppercase , could that be the error ?
And, yes, definitely saved off the original system keyspaces.
I'm tailing the logs when running the cassandra-cli, but I do not
see anything in the logs ..
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Thu, 11 Jul 2013 11:07:55 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Thu, Jul 11, 2013 at 11:00 AM, Langston, Jim <Ji...@compuware.com>> wrote:
I went through the whole sequence again and now have gotten to the point of
being able to try and pull in the schema, but now getting this error from the one
node I'm executing on.
[default@unknown] create keyspace OTracker
9209ec36-3b3f-3e24-9dfb-8a45a5b29a2a
Waiting for schema agreement...
... schemas agree across the cluster
NotFoundException()
This is pretty unusual.
All the nodes see each other and are available, all only contain a system
schema, none have a OTracker schema
If you look in the logs for schema related stuff when you try to create OTracker, what do you see?
Do you see the above UUID schema version in the logs?
At this point I am unable to suggest anything other than upgrading to the head of 1.1 line and try to create your keyspace there. There should be no chance of old state being implicated in your now stuck schema, so it seems likely that the problem has re-occured due to the version of Cassandra you are running.
Sorry I am unable to be of more assistance and that my advice appears to have resulted in your cluster being in worse condition than when you started. I probably mentioned but will do so again that if you have the old system keyspace directories, you can stop cassandra on all nodes and then revert to them.
=Rob
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Jul 11, 2013 at 11:00 AM, Langston, Jim
<Ji...@compuware.com>wrote:
> I went through the whole sequence again and now have gotten to the point
> of
> being able to try and pull in the schema, but now getting this error from
> the one
> node I'm executing on.
> [default@unknown] create keyspace OTracker
> 9209ec36-3b3f-3e24-9dfb-8a45a5b29a2a
> Waiting for schema agreement...
> ... schemas agree across the cluster
> NotFoundException()
>
This is pretty unusual.
> All the nodes see each other and are available, all only contain a system
> schema, none have a OTracker schema
>
If you look in the logs for schema related stuff when you try to create
OTracker, what do you see?
Do you see the above UUID schema version in the logs?
At this point I am unable to suggest anything other than upgrading to the
head of 1.1 line and try to create your keyspace there. There should be no
chance of old state being implicated in your now stuck schema, so it seems
likely that the problem has re-occured due to the version of Cassandra you
are running.
Sorry I am unable to be of more assistance and that my advice appears to
have resulted in your cluster being in worse condition than when you
started. I probably mentioned but will do so again that if you have the old
system keyspace directories, you can stop cassandra on all nodes and then
revert to them.
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
Thanks Rob,
I went through the whole sequence again and now have gotten to the point of
being able to try and pull in the schema, but now getting this error from the one
node I'm executing on.
[default@unknown] create keyspace OTracker
... with placement_strategy = 'SimpleStrategy'
... and strategy_options = {replication_factor : 3}
... and durable_writes = true;
9209ec36-3b3f-3e24-9dfb-8a45a5b29a2a
Waiting for schema agreement...
... schemas agree across the cluster
NotFoundException()
[default@unknown]
All the nodes see each other and are available, all only contain a system
schema, none have a OTracker schema
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Thu, 11 Jul 2013 10:35:43 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Thu, Jul 11, 2013 at 10:16 AM, Langston, Jim <Ji...@compuware.com>> wrote:
It feels like the 2 node that are not coming up with
the right schema are not seeing the nodes with the correct ones.
At the time that the nodes come up, they should have no schema other than the system columnfamilies. Only once all 3 nodes see each other should you be re-creating the schema. I'm not understanding your above sentence in light of this?
=Rob
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Jul 11, 2013 at 10:16 AM, Langston, Jim
<Ji...@compuware.com>wrote:
> It feels like the 2 node that are not coming up with
> the right schema are not seeing the nodes with the correct ones.
>
At the time that the nodes come up, they should have no schema other than
the system columnfamilies. Only once all 3 nodes see each other should you
be re-creating the schema. I'm not understanding your above sentence in
light of this?
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
Yes, I got the gist of what you were after, even making sure I broke
out the schema dump and load them in individually, but I haven't
gotten that far. It feels like the 2 node that are not coming up with
the right schema are not seeing the nodes with the correct ones.
And yes, I hear the beat of the upgrade drum, I was hoping to
do one step at a time so I don't carry my problem over.
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Thu, 11 Jul 2013 09:43:43 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Thu, Jul 11, 2013 at 9:17 AM, Langston, Jim <Ji...@compuware.com>> wrote:
Are the schema's held somewhere else ? Going through the
process that you sent, when I restart the nodes, the original
schema's show up
If you do not stop all nodes at once and then remove the system CFs, the existing schema will re-propogate via Gossip.
To be clear, I was suggesting that you dump the schema with cassandra-cli, erase the current schema with the cluster down, bring the cluster back up (NOW WITH NO SCHEMA) and then load the schema from the dump via cassandra-cli.
Also, in case I didn't mention it before, you should upgrade your version of Cassandra ASAP. :)
=Rob
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Jul 11, 2013 at 9:17 AM, Langston, Jim
<Ji...@compuware.com>wrote:
> Are the schema's held somewhere else ? Going through the
> process that you sent, when I restart the nodes, the original
> schema's show up
>
If you do not stop all nodes at once and then remove the system CFs, the
existing schema will re-propogate via Gossip.
To be clear, I was suggesting that you dump the schema with cassandra-cli,
erase the current schema with the cluster down, bring the cluster back up
(NOW WITH NO SCHEMA) and then load the schema from the dump via
cassandra-cli.
Also, in case I didn't mention it before, you should upgrade your version
of Cassandra ASAP. :)
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
Hi Rob,
Are the schema's held somewhere else ? Going through the
process that you sent, when I restart the nodes, the original
schema's show up (btw, you were correct on your assessment,
even though the schema shows they are the same with the
gossipinfo command, they are not the same when looking
at them with cassandra-cli, not even close on 2 of the nodes).
So, I went through the process of clearing out the system CF's,
in steps 4 and 5, when the cassandra's restarted two of them (the
ones with the incorrect schema's), complained about the schema
and loaded what looks like a generic one. But, all of them have
schemas and 2 are correct and one is not.
This means I cannot execute step 7 , since the schema now exists
with the name on all the nodes. For example, the incorrect schema is
called MySchema, after the restart and the messages complaining
about CF's not existing, there is a schema called MySchema, on 2 nodes
they are correct, on 2 nodes they are not.
I have also tried to force the node with the incorrect schema to come
up on its own by shutting down the cluster except for a node with a
correct schema. I went through the same steps and brought that
node down and back up, same results.
Thoughts ? ideas ?
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Tue, 9 Jul 2013 17:10:53 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Tue, Jul 9, 2013 at 11:52 AM, Langston, Jim <Ji...@compuware.com>> wrote:
>
> On the command (4 node cluster):
>
> nodetool gossipinfo -h localhost |grep SCHEMA |sort | uniq -c | sort -n
> 4 SCHEMA:60edeaa8-70a4-3825-90a5-d7746ffa8e4d
If your schemas actually agree (and given that you're in 1.1.2) you probably are encountering :
https://issues.apache.org/jira/browse/CASSANDRA-4432
Which is one of the 1.1.2 era "schema stuck" issues I was referring to earlier.
> On the second part, I have the same Cassandra version in staging and
> production, with staging being a smaller cluster. Not sure what you mean
> by nuking schema's (ie. delete directories ?)
I like when googling things returns related threads in which I have previously advised people to do a detailed list of things, heh :
http://mail-archives.apache.org/mod_mbox/cassandra-user/201208.mbox/%3CCAN1VBD-01aD7wT2w1eyY2KpHwcj+CoMjvE4=J5ZaSwYbmw_AyA@mail.gmail.com%3E
Here's a slightly clarified version of these steps...
0. Dump your existing schema to schema_definition_file
1. Take all nodes out of service;
2. Run nodetool drain on each and verify that they have drained (grep -i DRAINED system.log)
3. Stop cassandra on each node;
4. Move /var/lib/cassandra/data/system out of the way
5. Move /var/lib/cassandra/saved_caches/system-* out of the way
6. Start all nodes;
7. cassandra-cli < schema_definition_file on one node only. (includes
create keyspace and create column familiy entries)
Note: you should not literally do this, you should break your schema_definition_file into individual statements and wait until schema agreement between each DDL statement.
8. Put the nodes back in service.
9. Done.
=Rob
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 9, 2013 at 11:52 AM, Langston, Jim <Ji...@compuware.com>
wrote:
>
> On the command (4 node cluster):
>
> nodetool gossipinfo -h localhost |grep SCHEMA |sort | uniq -c | sort -n
> 4 SCHEMA:60edeaa8-70a4-3825-90a5-d7746ffa8e4d
If your schemas actually agree (and given that you're in 1.1.2) you
probably are encountering :
https://issues.apache.org/jira/browse/CASSANDRA-4432
Which is one of the 1.1.2 era "schema stuck" issues I was referring to
earlier.
> On the second part, I have the same Cassandra version in staging and
> production, with staging being a smaller cluster. Not sure what you mean
> by nuking schema's (ie. delete directories ?)
I like when googling things returns related threads in which I have
previously advised people to do a detailed list of things, heh :
http://mail-archives.apache.org/mod_mbox/cassandra-user/201208.mbox/%3CCAN1VBD-01aD7wT2w1eyY2KpHwcj+CoMjvE4=J5ZaSwYbmw_AyA@mail.gmail.com%3E
Here's a slightly clarified version of these steps...
0. Dump your existing schema to schema_definition_file
1. Take all nodes out of service;
2. Run nodetool drain on each and verify that they have drained (grep -i
DRAINED system.log)
3. Stop cassandra on each node;
4. Move /var/lib/cassandra/data/system out of the way
5. Move /var/lib/cassandra/saved_caches/system-* out of the way
6. Start all nodes;
7. cassandra-cli < schema_definition_file on one node only. (includes
create keyspace and create column familiy entries)
Note: you should not literally do this, you should break your
schema_definition_file into individual statements and wait until schema
agreement between each DDL statement.
8. Put the nodes back in service.
9. Done.
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
On the command (4 node cluster):
nodetool gossipinfo -h localhost |grep SCHEMA |sort | uniq -c | sort -n
4 SCHEMA:60edeaa8-70a4-3825-90a5-d7746ffa8e4d
On the second part, I have the same Cassandra version in staging and
production, with staging being a smaller cluster. Not sure what you mean
by nuking schema's (ie. delete directories ?)
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Tue, 9 Jul 2013 11:35:35 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Tue, Jul 9, 2013 at 10:26 AM, Robert Coli <rc...@eventbrite.com>> wrote:
nodetool -h localhost netstats |grep SCHEMA |sort | uniq -c | sort -n
Sorry, I meant "gossipinfo" and not "netstats".
With the right command, do you see that all nodes in the cluster have the same schema version?
I'm on version 1.1.2
1) Hinted Handoff is broken in 1.1.2, upgrade ASAP.
2) I believe the particular case you are encountering may be a more specific bug from the 1.1.2 timeframe.
3) Desynched schema like you seem to be encountering is very common in 1.1.2 timeframe. In most cases the best/only solution is :
a) drain all nodes, and stop them
b) nuke schema on all nodes (optionally nuke/move aside entire system keyspace)
c) start nodes, waiting for cluster to completely coalesce
d) re-load schema one statement at a time, BEING SURE TO WAIT FOR SCHEMA AGREEMENT ON ***ALL NODES*** before running the next schema altering statement
If you are unable to take an outage on this cluster, there are other ways to resolve issues like this but generally they will both be complex and error prone and will take much more time and effort than doing the above.
=Rob
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 9, 2013 at 10:26 AM, Robert Coli <rc...@eventbrite.com> wrote:
> nodetool -h localhost netstats |grep SCHEMA |sort | uniq -c | sort -n
>
Sorry, I meant "gossipinfo" and not "netstats".
With the right command, do you see that all nodes in the cluster have the
same schema version?
> I'm on version 1.1.2
>
1) Hinted Handoff is broken in 1.1.2, upgrade ASAP.
2) I believe the particular case you are encountering may be a more
specific bug from the 1.1.2 timeframe.
3) Desynched schema like you seem to be encountering is very common in
1.1.2 timeframe. In most cases the best/only solution is :
a) drain all nodes, and stop them
b) nuke schema on all nodes (optionally nuke/move aside entire system
keyspace)
c) start nodes, waiting for cluster to completely coalesce
d) re-load schema one statement at a time, BEING SURE TO WAIT FOR SCHEMA
AGREEMENT ON ***ALL NODES*** before running the next schema altering
statement
If you are unable to take an outage on this cluster, there are other ways
to resolve issues like this but generally they will both be complex and
error prone and will take much more time and effort than doing the above.
=Rob
Re: alter column family ?
Posted by "Langston, Jim" <Ji...@compuware.com>.
I'm on version 1.1.2
The nodetool command by itself
# nodetool netstats -h localhost
Mode: NORMAL
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 5909
Responses n/a 0 1203
Jim
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: <us...@cassandra.apache.org>>
Date: Tue, 9 Jul 2013 10:26:45 -0700
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: alter column family ?
On Tue, Jul 9, 2013 at 10:08 AM, Langston, Jim <Ji...@compuware.com>> wrote:
I am trying to alter a column family to change gc_grace_seconds, and now,
any of the properties
The sequence:
use ks ;
alter table CF with gc_grace_seconds=864000 ;
When listing the CF, gc_grace_seconds is set to 0, after
running the CLI, gc_grace_seconds is still set to 0.
I tried change the comment property, but this did not
change either.
Using the same keyspace, I created another table
and executed both the gc_grace_seconds change and
the comments change. Both of these successfully changed.
I'm surprised to hear that other columnfamilies work in the same keyspace. In general when people end up in this case, no schema migrations work.
If you do :
nodetool -h localhost netstats |grep SCHEMA |sort | uniq -c | sort -n
Do you see that all nodes in the cluster have the same schema version?
What version of Cassandra?
=Rob
Re: alter column family ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 9, 2013 at 10:08 AM, Langston, Jim
<Ji...@compuware.com>wrote:
> I am trying to alter a column family to change gc_grace_seconds, and
> now,
> any of the properties
>
> The sequence:
>
> use ks ;
> alter table CF with gc_grace_seconds=864000 ;
>
> When listing the CF, gc_grace_seconds is set to 0, after
> running the CLI, gc_grace_seconds is still set to 0.
>
> I tried change the comment property, but this did not
> change either.
>
> Using the same keyspace, I created another table
> and executed both the gc_grace_seconds change and
> the comments change. Both of these successfully changed.
>
I'm surprised to hear that other columnfamilies work in the same keyspace.
In general when people end up in this case, no schema migrations work.
If you do :
nodetool -h localhost netstats |grep SCHEMA |sort | uniq -c | sort -n
Do you see that all nodes in the cluster have the same schema version?
What version of Cassandra?
=Rob