Upgrade from 1.2.19 to 2.0.12 -- seeing lots of SliceQueryFilter messages in system.log

["Caraballo, Rafael" <ra...@twcable.com>]

After upgrading a 3 node Cassandra cluster from 1.2.19 to 2.0.12, I have an event storm of " SliceQueryFilter" messages flooding the Cassandra system.log file.

WARN [ReadStage:1043] 2015-03-18 15:14:12,708 SliceQueryFilter.java (line 231) Read 201 live and 13539 tombstoned cells in KeyspaceMetadata.CF_Folder (see tombstone_warn_threshold). 200 columns was requested, slices=[154184c2-85c1-11e2-b12e-c2ed2ac02b21-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647, ranges=[dc70cafe-ed8a-11e2-a178-5756012ec923-dc70cafe-ed8a-11e2-a178-5756012ec923:!, deletedAt=1424741296925196, localDeletion=1424741340][82bcb57a-ed8c-11e2-8fbd-3fb065c6b097-82bcb57a-ed8c-11e2-8fbd-3fb065c6b097:!, deletedAt=1424741296925196,...

This is the table definition referenced above:

CREATE TABLE "CF_Folder" (
  key blob,
  column1 timeuuid,
  column2 blob,
  value blob,
  PRIMARY KEY ((key), column1, column2)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.100000 AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=518400 AND
  read_repair_chance=0.100000 AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  compaction={'sstable_size_in_mb': '160', 'class': 'LeveledCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

How can I stop this event storm?

Thanks,
Rafael Caraballo
Time Warner Cable

________________________________
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

Re: Upgrade from 1.2.19 to 2.0.12 -- seeing lots of SliceQueryFilter messages in system.log

[Robert Coli <rc...@eventbrite.com>]

On Wed, Mar 18, 2015 at 10:14 AM, Caraballo, Rafael <
rafael.caraballo@twcable.com> wrote:

>  After upgrading a 3 node Cassandra cluster from 1.2.19 to 2.0.12, I have
> an event storm of “ SliceQueryFilter” messages flooding the Cassandra
> system.log file.
>
>
>
> How can I stop this event storm?
>
>
>
As the message says :

" (see tombstone_warn_threshold) "

The thing you are being warned about is that your write pattern results in
a significant number of tombstones. In general this is a smell of badness
in Cassandra, which is why the log message exists.

To Resolve :

1) Increase tombstone_warn_threshold
OR
2) Stop creating so many tombstones

=Rob
http://twitter.com/rcolidba