You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anshul Rathore <an...@gmail.com> on 2018/07/04 18:08:33 UTC
Auto Compactions not running on Cassandra 3.10
Hi
We are using 4 node cassandra 3.10 cluster.
For some reason autocompaction is not running on one of the table which
uses DTCS. TTL on this table is 3 months. Table has high write load , and
medium read load.
We have 4 disks per node , each disk growed to around 5k-6k sstables going
back to around 6-7 months.
I tried running major compaction which takes 1-2 days on each node, which
results in 20-30 sstables on all disks.
But also after running major compaction , we now see sstables have started
growing again. So minor/auto compaction is not running on this table.
*Few Observations*
* - *There are 2-3 partition keys i see while doing major compaction
throwing large partition warning(100mb threshold) , their sizes around
100-200MB
- All other tables use STCS , their auto compaction is working fine.
Any pointers would be helpful
Thanks,
Anshul
Re: Auto Compactions not running on Cassandra 3.10
Posted by Anshul Rathore <an...@gmail.com>.
Hi
It would be great if anyone can point us in right direction.
On Mon, Jul 30, 2018 at 12:07 PM Anshul Rathore <an...@gmail.com>
wrote:
> Thanks Jeff for your response ,and apologies for such a delayed response ,
> had some personal emergency.
>
> So Following the config which we are using for that table
> PRIMARY KEY ((customer_app_prefix, customer_session_id),
> beacon_client_type, sim_created_at)
> ) WITH CLUSTERING ORDER BY (beacon_client_type ASC, sim_created_at DESC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
>
> I didn't get the part *"are you using milliseconds or microseconds in
> your writes?" *,
> did you mean for the timestamp column in Primary Key , for that we using
> milliseconds .
>
> Thanks,
> Anshul
>
>
> On Wed, Jul 4, 2018 at 11:55 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> The DTCS windowing algorithm is timestamp sensitive - are you using
>> milliseconds or microseconds in your writes? What’s your exact DTCS config
>> for that table?
>>
>> 3.10 has TWCS, too, which may be considerably easier to use. It also uses
>> timestamp units in its config, so you’ll need to know which resolution
>> you’re using on writes
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Jul 4, 2018, at 11:08 AM, Anshul Rathore <an...@gmail.com>
>> wrote:
>>
>>
>> Hi
>>
>> We are using 4 node cassandra 3.10 cluster.
>> For some reason autocompaction is not running on one of the table which
>> uses DTCS. TTL on this table is 3 months. Table has high write load , and
>> medium read load.
>>
>> We have 4 disks per node , each disk growed to around 5k-6k sstables
>> going back to around 6-7 months.
>> I tried running major compaction which takes 1-2 days on each node, which
>> results in 20-30 sstables on all disks.
>> But also after running major compaction , we now see sstables have
>> started growing again. So minor/auto compaction is not running on this
>> table.
>>
>> *Few Observations*
>> * - *There are 2-3 partition keys i see while doing major
>> compaction throwing large partition warning(100mb threshold) , their sizes
>> around 100-200MB
>> - All other tables use STCS , their auto compaction is working
>> fine.
>>
>> Any pointers would be helpful
>>
>> Thanks,
>> Anshul
>>
>>
>
> --
> *Regards,*
> *Anshul Rathore*
>
--
*Regards,*
*Anshul Rathore*
Re: Auto Compactions not running on Cassandra 3.10
Posted by Anshul Rathore <an...@gmail.com>.
Thanks Jeff for your response ,and apologies for such a delayed response ,
had some personal emergency.
So Following the config which we are using for that table
PRIMARY KEY ((customer_app_prefix, customer_session_id),
beacon_client_type, sim_created_at)
) WITH CLUSTERING ORDER BY (beacon_client_type ASC, sim_created_at DESC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
I didn't get the part *"are you using milliseconds or microseconds in your
writes?" *,
did you mean for the timestamp column in Primary Key , for that we using
milliseconds .
Thanks,
Anshul
On Wed, Jul 4, 2018 at 11:55 PM Jeff Jirsa <jj...@gmail.com> wrote:
> The DTCS windowing algorithm is timestamp sensitive - are you using
> milliseconds or microseconds in your writes? What’s your exact DTCS config
> for that table?
>
> 3.10 has TWCS, too, which may be considerably easier to use. It also uses
> timestamp units in its config, so you’ll need to know which resolution
> you’re using on writes
>
>
> --
> Jeff Jirsa
>
>
> On Jul 4, 2018, at 11:08 AM, Anshul Rathore <an...@gmail.com>
> wrote:
>
>
> Hi
>
> We are using 4 node cassandra 3.10 cluster.
> For some reason autocompaction is not running on one of the table which
> uses DTCS. TTL on this table is 3 months. Table has high write load , and
> medium read load.
>
> We have 4 disks per node , each disk growed to around 5k-6k sstables going
> back to around 6-7 months.
> I tried running major compaction which takes 1-2 days on each node, which
> results in 20-30 sstables on all disks.
> But also after running major compaction , we now see sstables have started
> growing again. So minor/auto compaction is not running on this table.
>
> *Few Observations*
> * - *There are 2-3 partition keys i see while doing major
> compaction throwing large partition warning(100mb threshold) , their sizes
> around 100-200MB
> - All other tables use STCS , their auto compaction is working fine.
>
> Any pointers would be helpful
>
> Thanks,
> Anshul
>
>
--
*Regards,*
*Anshul Rathore*
Re: Auto Compactions not running on Cassandra 3.10
Posted by Jeff Jirsa <jj...@gmail.com>.
The DTCS windowing algorithm is timestamp sensitive - are you using milliseconds or microseconds in your writes? What’s your exact DTCS config for that table?
3.10 has TWCS, too, which may be considerably easier to use. It also uses timestamp units in its config, so you’ll need to know which resolution you’re using on writes
--
Jeff Jirsa
> On Jul 4, 2018, at 11:08 AM, Anshul Rathore <an...@gmail.com> wrote:
>
>
> Hi
>
> We are using 4 node cassandra 3.10 cluster.
> For some reason autocompaction is not running on one of the table which uses DTCS. TTL on this table is 3 months. Table has high write load , and medium read load.
>
> We have 4 disks per node , each disk growed to around 5k-6k sstables going back to around 6-7 months.
> I tried running major compaction which takes 1-2 days on each node, which results in 20-30 sstables on all disks.
> But also after running major compaction , we now see sstables have started growing again. So minor/auto compaction is not running on this table.
>
> Few Observations
> - There are 2-3 partition keys i see while doing major compaction throwing large partition warning(100mb threshold) , their sizes around 100-200MB
> - All other tables use STCS , their auto compaction is working fine.
>
> Any pointers would be helpful
>
> Thanks,
> Anshul