Previously deleted rows resurrected by repair?

[Jonas Borgström <jo...@borgstrom.se>]

Hi,

I Have a 3 node cluster running Cassandra 1.0.3 and using replication 
factor=3.

Recently I've noticed that some previously deleted rows have started to 
reappear for some reason. And now I wonder if this is a known issue with 
1.0.3?

Repairs have been running every weekend (gc_grace is 10 days) and always 
completed successfully. But while looking at the logs I noticed that a 
fair number of ranges (around 10% of the total number of keys) have been 
streamed between these nodes during the repair sessions. This seems a 
bit high to me given that everything is written using quorum and all 
nodes have been up all the time.

For me this looks suspiciously like some already deleted keys are 
streamed to other nodes during repair.


Some more details about the data:
All keys are written to only once and most of them are deleted a couple 
of days/weeks later. Some keys are large enough to require incremental 
compaction.

Could this bug cause this?

https://issues.apache.org/jira/browse/CASSANDRA-3510

Regards,
Jonas

Re: Previously deleted rows resurrected by repair?

[Edward Capriolo <ed...@gmail.com>]

I wanted to throw out a "solution" of mine . With hinted handoff being
guaranteed losing a delete is almost impossible so solutions like this are
no longer required, but...

My "solution" is always write deletes at cl.all. if a delete fails at all
queue it up, on a message queue, write a log, etc. Then later on have some
process consume the log and replay the write the delete at all again.

This sounds like a pain but it it really not that bad. if you are using c*
you likely already have a message queue or hadoop system. Generally lost
deletes is an edge case so you would not expect this backup system to be
very busy.


On Wednesday, December 28, 2011, Jonas Borgström <jo...@borgstrom.se> wrote:
> On 2011-12-28 12:52 , Dominic Williams wrote:
>>
>> Hmm interesting could be some variation on 3510 (which caught me out).
>
> Actually after making some further reading of the changelog 2786 looks
like a likely culprit.
> If I'm reading the jira correctly all versions < 0.8.8 and < 1.0.4 are at
risk of getting deleted rows resurrected.
>
> Hopefully an upgrade to 1.0.6 will stop this problem from increasing but
I still need to manually re-delete a bunch of rows.
>
> https://issues.apache.org/jira/browse/CASSANDRA-2786
>
>> Personally I really don't like having to rely on repair to stop deletes
>> being undone. If you agree follow this proposal for an alternative
>> https://issues.apache.org/jira/browse/CASSANDRA-3620 which also stops
>> tombstone build up.
>
> Thanks, this definitely looks interesting. I'll have a look.
>
> / Jonas
>

Re: Previously deleted rows resurrected by repair?

[Jonas Borgström <jo...@borgstrom.se>]

On 2011-12-28 12:52 , Dominic Williams wrote:
> Hmm interesting could be some variation on 3510 (which caught me out).

Actually after making some further reading of the changelog 2786 looks 
like a likely culprit.
If I'm reading the jira correctly all versions < 0.8.8 and < 1.0.4 are 
at risk of getting deleted rows resurrected.

Hopefully an upgrade to 1.0.6 will stop this problem from increasing but 
I still need to manually re-delete a bunch of rows.

https://issues.apache.org/jira/browse/CASSANDRA-2786

> Personally I really don't like having to rely on repair to stop deletes
> being undone. If you agree follow this proposal for an alternative
> https://issues.apache.org/jira/browse/CASSANDRA-3620 which also stops
> tombstone build up.

Thanks, this definitely looks interesting. I'll have a look.

/ Jonas

Re: Previously deleted rows resurrected by repair?

[Dominic Williams <dw...@fightmymonster.com>]

Hmm interesting could be some variation on 3510 (which caught me out).

Personally I really don't like having to rely on repair to stop deletes
being undone. If you agree follow this proposal for an alternative
https://issues.apache.org/jira/browse/CASSANDRA-3620 which also stops
tombstone build up.

2011/12/27 Jonas Borgström <jo...@borgstrom.se>

> Hi,
>
> I Have a 3 node cluster running Cassandra 1.0.3 and using replication
> factor=3.
>
> Recently I've noticed that some previously deleted rows have started to
> reappear for some reason. And now I wonder if this is a known issue with
> 1.0.3?
>
> Repairs have been running every weekend (gc_grace is 10 days) and always
> completed successfully. But while looking at the logs I noticed that a fair
> number of ranges (around 10% of the total number of keys) have been
> streamed between these nodes during the repair sessions. This seems a bit
> high to me given that everything is written using quorum and all nodes have
> been up all the time.
>
> For me this looks suspiciously like some already deleted keys are streamed
> to other nodes during repair.
>
>
> Some more details about the data:
> All keys are written to only once and most of them are deleted a couple of
> days/weeks later. Some keys are large enough to require incremental
> compaction.
>
> Could this bug cause this?
>
> https://issues.apache.org/**jira/browse/CASSANDRA-3510<https://issues.apache.org/jira/browse/CASSANDRA-3510>
>
> Regards,
> Jonas
>