You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kai Wang <de...@gmail.com> on 2015/12/08 00:46:57 UTC

lots of tombstone after compaction

I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data are
large amount of wide rows with lots of null's. It takes one day or two for
the compaction to complete. sstable count is at single digit. Maximum
partition size is ~50M and mean size is ~5M. However I am seeing frequent
read query timeouts caused by tombstone_failure_threshold (100000). These
tables are basically read-only. There're no writes.

I just kicked off compaction on those tables using nodetool. Hopefully it
can remove those tombstones. But is it normal to have these many tombstones
after the initial compactions? Is this related to the fact the original
data has lots of nulls?

Thanks.

Re: lots of tombstone after compaction

Posted by Kai Wang <de...@gmail.com>.
Rob and Jeff,

Thank you. It makes sense. I am on 2.1.10 and will upgrade to 2.1.12.
On Dec 7, 2015 7:05 PM, "Jeff Jirsa" <je...@crowdstrike.com> wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-7953
>
> https://issues.apache.org/jira/browse/CASSANDRA-10505
>
> There are buggy versions of cassandra that will multiple tombstones during
> compaction. 2.1.12 SHOULD correct that, if you’re on 2.1.
>
>
>
> From: Kai Wang
> Reply-To: "user@cassandra.apache.org"
> Date: Monday, December 7, 2015 at 3:46 PM
> To: "user@cassandra.apache.org"
> Subject: lots of tombstone after compaction
>
> I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data
> are large amount of wide rows with lots of null's. It takes one day or two
> for the compaction to complete. sstable count is at single digit. Maximum
> partition size is ~50M and mean size is ~5M. However I am seeing frequent
> read query timeouts caused by tombstone_failure_threshold (100000). These
> tables are basically read-only. There're no writes.
>
> I just kicked off compaction on those tables using nodetool. Hopefully it
> can remove those tombstones. But is it normal to have these many tombstones
> after the initial compactions? Is this related to the fact the original
> data has lots of nulls?
>
> Thanks.
>

Re: lots of tombstone after compaction

Posted by Jeff Jirsa <je...@crowdstrike.com>.
https://issues.apache.org/jira/browse/CASSANDRA-7953

https://issues.apache.org/jira/browse/CASSANDRA-10505

There are buggy versions of cassandra that will multiple tombstones during compaction. 2.1.12 SHOULD correct that, if you’re on 2.1.



From:  Kai Wang
Reply-To:  "user@cassandra.apache.org"
Date:  Monday, December 7, 2015 at 3:46 PM
To:  "user@cassandra.apache.org"
Subject:  lots of tombstone after compaction

I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data are large amount of wide rows with lots of null's. It takes one day or two for the compaction to complete. sstable count is at single digit. Maximum partition size is ~50M and mean size is ~5M. However I am seeing frequent read query timeouts caused by tombstone_failure_threshold (100000). These tables are basically read-only. There're no writes.

I just kicked off compaction on those tables using nodetool. Hopefully it can remove those tombstones. But is it normal to have these many tombstones after the initial compactions? Is this related to the fact the original data has lots of nulls?

Thanks.

Re: lots of tombstone after compaction

Posted by Robert Wille <rw...@fold3.com>.
The nulls in the original data created the tombstones. They won’t go away until gc_grace_seconds have passed (default is 10 days).

On Dec 7, 2015, at 4:46 PM, Kai Wang <de...@gmail.com> wrote:

> I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data are large amount of wide rows with lots of null's. It takes one day or two for the compaction to complete. sstable count is at single digit. Maximum partition size is ~50M and mean size is ~5M. However I am seeing frequent read query timeouts caused by tombstone_failure_threshold (100000). These tables are basically read-only. There're no writes.
> 
> I just kicked off compaction on those tables using nodetool. Hopefully it can remove those tombstones. But is it normal to have these many tombstones after the initial compactions? Is this related to the fact the original data has lots of nulls?
> 
> Thanks.