How do I repair what looks to be a corrupt sstable index?

[Scott McCarty <sc...@metajinx.com>]

Hi,

I don't know how data on at least one of our five nodes in our cluster got
bad but what I'm seeing is that the key value output from sstable2json
doesn't match the key value that I specify on the command line.

For example, I do the following on one of the nodes:

bash$:/root Mon Feb 14 12:47:30pm
===> . /opt/cassandra/config/core/cassandra.in.sh &&
/opt/cassandra/bin/sstable2json
 /opt/cassandra/storage/core/data/p1/cf1-6239-Data.db -k
US15600be0-e8b4-4439-a7df-d2e21aace7c4:all
2011-02-14 12:47:47,315 INFO
[org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto'
determined to be mmap, indexAccessMode is mmap
2011-02-14 12:47:47,951 INFO [org.apache.cassandra.io.SSTableReader] -
Opening /opt/cassandra/storage/core/data/p1/cf1-6239-Data.db
{
  "USc5494dfa-678a-4175-a1d5-65730065c69d:UNHIDE":
[["25636f6e763a50524331373331663131662d383466342d343463662d393037652d343565393732326561346161",
"05f49d9d252e010000", 1297711209972000, false],
["256d6f64646174653a313239373730383732323632343a50524331373331663131662d383466342d343463662d393037652d343565393732326561346161",
"4d597f9f", 1297710990061000,
true],
["256d6f64646174653a313239373731303939303036313a50524331373331663131662d383466342d343463662d393037652d343565393732326561346161",
"4d59806b", 1297711209972000, true],
["256d6f64646174653a313239373731313230393937323a50524331373331663131662d383466342d343463662d393037652d343565393732326561346161",
"04555337303266363235652d643933362d343138342d613834392d376330616635323430396165",
1297711209972000, false]]
}

Notice that the key I specified on the command line
("US15600be0-e8b4-4439-a7df-d2e21aace7c4:all") doesn't match what comes back
as the JSON key ("USc5494dfa-678a-4175-a1d5-65730065c69d:UH").  These match
on a small sample of other nodes and other keys (as I would expect).

If this is a corrupt sstable index, how can I repair this?  We're running
0.6.11 and have RF=3 with the 5 nodes and have been using CL=QUORUM for
reads and writes for quite a while.  Something tells me it's not as easy as
deleting the index file :-)

Of course, the bigger question is how did it get this way...

Thanks!
  Scott