You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Xu Zhongxing <xu...@163.com> on 2015/01/13 05:27:01 UTC

nodetool compact cannot remove tombstone in system keyspace

Hi,


When I connect to C* with driver, I found some warnings in the log (I increased tombstone_failure_threshold to 150000 to see the warning)


WARN [ReadStage:5] 2015-01-13 12:21:14,595 SliceQueryFilter.java (line 225) Read 34188 live and 104186 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147483387 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
 WARN [ReadStage:5] 2015-01-13 12:21:15,562 SliceQueryFilter.java (line 225) Read 34209 live and 104247 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147449199 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}


I run the command:
nodetool compact system 


But the tombstone number does not decrease. I still see the warnings with the exact number of tombstones.
Why is this happening? What should I do to remove the tombstones in the system keyspace?

Re: nodetool compact cannot remove tombstone in system keyspace

Posted by Xu Zhongxing <xu...@163.com>.

Thanks for confirming "the tombstones will only get removed during compaction if they are older than GC_Grace_Seconds for that CF". I didn't find such a clarification in the documentation. That answered my question.


Since the table that has too many tombstones is in the system keyspace, I cannot alter its gc_grace_seconds setting. gc_grace_seconds is now 7 days, which is certainly longer than the age of the tombstones. 


Is there any way that I can remove the tombstones in the system keyspace immediately?

At 2015-01-13 19:49:47, "Rahul Neelakantan" <ra...@rahul.be> wrote:

I am not sure about the tombstone_failure_threshold, but the tombstones will only get removed during compaction if they are older than GC_Grace_Seconds for that CF. How old are these tombstones?

Rahul

On Jan 12, 2015, at 11:27 PM, Xu Zhongxing <xu...@163.com> wrote:


Hi,


When I connect to C* with driver, I found some warnings in the log (I increased tombstone_failure_threshold to 150000 to see the warning)


WARN [ReadStage:5] 2015-01-13 12:21:14,595 SliceQueryFilter.java (line 225) Read 34188 live and 104186 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147483387 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
 WARN [ReadStage:5] 2015-01-13 12:21:15,562 SliceQueryFilter.java (line 225) Read 34209 live and 104247 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147449199 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}


I run the command:
nodetool compact system 


But the tombstone number does not decrease. I still see the warnings with the exact number of tombstones.
Why is this happening? What should I do to remove the tombstones in the system keyspace?

Re: nodetool compact cannot remove tombstone in system keyspace

Posted by Rahul Neelakantan <ra...@rahul.be>.

I am not sure about the tombstone_failure_threshold, but the tombstones will only get removed during compaction if they are older than GC_Grace_Seconds for that CF. How old are these tombstones?

Rahul

> On Jan 12, 2015, at 11:27 PM, Xu Zhongxing <xu...@163.com> wrote:
> 
> Hi,
> 
> When I connect to C* with driver, I found some warnings in the log (I increased tombstone_failure_threshold to 150000 to see the warning)
> 
> WARN [ReadStage:5] 2015-01-13 12:21:14,595 SliceQueryFilter.java (line 225) Read 34188 live and 104186 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147483387 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
>  WARN [ReadStage:5] 2015-01-13 12:21:15,562 SliceQueryFilter.java (line 225) Read 34209 live and 104247 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147449199 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
> 
> I run the command:
> nodetool compact system 
> 
> But the tombstone number does not decrease. I still see the warnings with the exact number of tombstones.
> Why is this happening? What should I do to remove the tombstones in the system keyspace?