Cassandra SSTables' metadata and corruption

[Rajath Subramanyam <ra...@gmail.com>]

Hello C*-users community,

I have a question about Cassandra data files corruption. In one of our test
clusters (which runs *C* 2.1*), we observed that when we run the tool
*'sstablemetada'* on one of the sstables, it returns an *invalid value for
minimum timestamp*:

[datos_db_user@sgp-src-1 pre-scrub-1464981676212]$ java -jar
sstablemetadata.jar vt1-table_1-ka-12-Data.db | grep "timestamp"
Minimum timestamp: -9223372036854775808
Maximum timestamp: 1464380906180000

From the sstablemetadata tool source code, I know that it parses only the
Statistics.db component to print the metadata. This seems to indicate three
possibilities:

   1. There is a corruption in the Statistics.db component only and the
   Data in the Data.db is fine
   2. There is a corruption in the data in the Data.db which is causing the
   invalid timestamp to show up in the Statistics.db component
   3. There is a bug in the sstablemetadata tool

However, when I run sstablescrub on the sstable, the problem is solved and
I get the right metadata. This rules out 3 above. Hence it is either 1 or
2.

[datos_db_user@sgp-src-1 table_1-99d6ca40244611e697fb176f47d4f7f7]$ java
-jar sstablemetadata.jar vt1-table_1-ka-16-Data.db | grep
"timestamp\|Ancestors"
Minimum timestamp: 1464380729962999
Maximum timestamp: 1464380906180000
Ancestors: [12]
[datos_db_user@sgp-src-1 table_1-99d6ca40244611e697fb176f47d4f7f7]$

So my question is, for those of you who have been running Cassandra for a
long time in production, how often does the Statistics.db component get
corrupted resulting in sstablemetadata showing invalid metadata ? Does
Cassandra guarantee that the metadata shown by sstablemetadata is reliable
? If the metadata being shown by sstablemetadata is invalid, does it point
to a bigger data corruption in the Data.db component ?

Thanks in advance for your thoughts about this.

Thank you !

Regards,
Rajath

------------------------
Rajath Subramanyam