Data loss after system_keyspace-table removal?

[Bastian Schnorbus <ba...@gmail.com>]

Hi experts!



Seems, that we have a data loss after a rolling update of our Cassandra
2.2.7 nodes.

The nodes failed to start with an exception:

IllegalStateException: empty rows returned when reading
system.schema_keyspaces à (
https://issues.apache.org/jira/browse/CASSANDRA-12351)




*Workaround mentioned in the ticket:*-
*Stop node , *-
*remove sstables in data/system/schema_keyspaces/*-          *restart.*

*      After a restart expect load to be higher for a short period.*



In our case the deleted system_keyspace-table is rebuild but it doesn’t
contain the data-keyspaces anymore!!! The data still seem to be on disk...



Did someone also encounter this issue and solved it using the above
workaround?



Do we need to add the meta-information for the existing keyspaces manually
to that table? Do we need to wait (according to the warning in the
workaround) for the tables/keyspaces to recover itself?

What can we do to get the keyspaces and data back?



Many thanks in advance,

Bastian