Can't write to row key, even at ALL. Tombstones?

[Josh Dzielak <jo...@keen.io>]

We have a few row keys that aren’t taking any writes, even using both the ALL consistency level to read and write. We can’t insert anything into any column, previously existing or new, using the simplest possible heuristic in cqlsh or cassandra-cli.  

Our suspicion is that we somehow have a row level tombstone that is future-dated and has not gone away (we’ve lowered gc_grace_seconds in hope that it’d get compacted, but no luck so far, even though the stables that hold the row key have all cycled since).

How can we make this row key writeable again? Or even better, what can we do to debug this? Is there a way to get Cassandra to log all of the read candidates, including timestamp and host and sstable, before it chooses the one to use?  

Thanks as always,
-Josh

Re: Can't write to row key, even at ALL. Tombstones?

[Edward Capriolo <ed...@gmail.com>]

The sstable to json tools can be used to remove keys from underlying files.
It requires restarts.

On Friday, December 27, 2013, Josh Dzielak <jo...@keen.io> wrote:
> We have a few row keys that aren’t taking any writes, even using both the
ALL consistency level to read and write. We can’t insert anything into any
column, previously existing or new, using the simplest possible heuristic
in cqlsh or cassandra-cli.
> Our suspicion is that we somehow have a row level tombstone that
is future-dated and has not gone away (we’ve lowered gc_grace_seconds in
hope that it’d get compacted, but no luck so far, even though the stables
that hold the row key have all cycled since).
> How can we make this row key writeable again? Or even better, what can we
do to debug this? Is there a way to get Cassandra to log all of the read
candidates, including timestamp and host and sstable, before it chooses the
one to use?
> Thanks as always,
> -Josh

-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: Can't write to row key, even at ALL. Tombstones?

[Russell Bradberry <rb...@gmail.com>]

That link is broken, try this one: http://thelastpickle.com/blog/2011/12/15/Anatomy-of-a-Cassandra-Partition.html



On December 30, 2013 at 2:02:15 PM, Robert Coli (rcoli@eventbrite.com) wrote:

On Fri, Dec 27, 2013 at 6:13 PM, Josh Dzielak <jo...@keen.io> wrote:
Our suspicion is that we somehow have a row level tombstone that is future-dated and has not gone away (we’ve lowered gc_grace_seconds in hope that it’d get compacted, but no luck so far, even though the stables that hold the row key have all cycled since).

What version of Cassandra?

ecapriolo is right, use sstablekeys/sstable2json to inspect suspect rows in SSTables.

If that's the problem, you can follow this procedure :


http://thelastpickle.com/2011/12/15/Anatomy-of-a-Cassandra-Partition/

Or you can dump/reload with sstable2json/json2sstable and filter out bad values.

=Rob

Re: Can't write to row key, even at ALL. Tombstones?

[Robert Coli <rc...@eventbrite.com>]

On Fri, Dec 27, 2013 at 6:13 PM, Josh Dzielak <jo...@keen.io> wrote:

> Our suspicion is that we somehow have a row level tombstone that
> is future-dated and has not gone away (we’ve lowered gc_grace_seconds in
> hope that it’d get compacted, but no luck so far, even though the stables
> that hold the row key have all cycled since).
>

What version of Cassandra?

ecapriolo is right, use sstablekeys/sstable2json to inspect suspect rows in
SSTables.

If that's the problem, you can follow this procedure :

http://thelastpickle.com/2011/12/15/Anatomy-of-a-Cassandra-Partition/

Or you can dump/reload with sstable2json/json2sstable and filter out bad
values.

=Rob