Steps to Manually Resolve Stuck Schema!! CASSANDRA-4561

[Arya Goudarzi <go...@gmail.com>]

Just had a good conversation with rcoli in chat. Wanted to clarify the
steps for resolving this issue and see if there are any pitfalls I am
missing.

Issue: I upgraded from 1.1.2 to 1.1.3 a while ago and today I realized I
cannot make any schema changes since the fix in
https://issues.apache.org/jira/browse/CASSANDRA-4432.

Solution: Somehow I have to make Cassandra system's column family to forget
about those old schemas with nanosecond timestamps. I have to do this
either live or with a brief downtime. Please advice of any pitfalls or
incorrectness in my steps. I am planning to automate them so please advice.

Within a short downtime, I have to do this:

1. Take all nodes out of service;
2. Run nodetool flush on each;
3. Stop cassandra on each node;
4. Remove /var/lib/cassandra/data/system
5. Remove /var/lib/cassandra/saved_caches/system-*
6. Start all nodes;
7. cassandra-cli < schema_definition_file on one node only. (includes
create keyspace and create column familiy entries)
8. put the nodes back in service.
9. Done.

Please advice if I have got the steps correctly or if I am missing
something.

Thanks in advance for you help.

Cheers,
-Arya