You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Keith Wright <kw...@nanigans.com> on 2013/07/22 15:02:22 UTC
sstable size change
Hi all,
I know there has been several threads recently on this but I wanted to make sure I got a clear answer: we are looking to increase our SSTable size for a couple of our LCS tables as well as chunk size (to match the SSD block size). The largest table is at 500 GB across 6 nodes (RF 3, C* 1.2.4 VNodes). I wanted to get feedback on the best way to make this change with minimal load impact on the cluster. After I make the change, I understand that I need to force the nodes to re-compact the tables.
Can this be done via upgrade sstables or do I need to shutdown the node, delete the .json file, and restart as some have suggested?
I assume I can do this one node at a time?
If I change the bloom filter size, I assume I will need to force compaction again? Using the same methodology?
Thank you
Re: sstable size change
Posted by Keith Wright <kw...@nanigans.com>.
Created https://issues.apache.org/jira/browse/CASSANDRA-5834
From: aaron morton <aa...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, July 31, 2013 5:11 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
Re: sstable size change
Posted by aaron morton <aa...@thelastpickle.com>.
Can you put that in a ticket ? https://issues.apache.org/jira/browse/CASSANDRA
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 30/07/2013, at 12:52 AM, Keith Wright <kw...@nanigans.com> wrote:
> Version 1.2.4.
>
> Original alter that does nothing: alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
>
> Correct alter: alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
>
> Thanks
>
> From: aaron morton <aa...@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Monday, July 29, 2013 4:02 AM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: sstable size change
>
>> I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
> Sounds like a good idea.
>
> Please describe the version and query you first used.
>
> Thanks for taking the time to update the thread.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/07/2013, at 12:57 AM, Keith Wright <kw...@nanigans.com> wrote:
>
>> FYI. It appears that the proper command in CQL3 is the following:
>>
>> alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
>>
>> I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
>>
>> alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
>>
>>
>> From: Keith Wright <kw...@nanigans.com>
>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Date: Thursday, July 25, 2013 9:14 AM
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>, Wei Zhu <wz...@yahoo.com>
>> Subject: Re: sstable size change
>>
>> Unfortunately the table in question is a CQL3 table so cassandra-cli will not output its describe:
>>
>> WARNING: CQL3 tables are intentionally omitted from 'describe' output.
>> See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
>>
>> However, I did figure out that apparently I was not setting the change properly. I was using the following alter via CQL3:
>>
>> alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
>>
>> But when I did a describe, I did not see the change.
>>
>> CREATE TABLE shard_user_lookup (
>> shard_user_id int,
>> shard text,
>> creation_time timestamp,
>> status int,
>> user_id timeuuid,
>> PRIMARY KEY (shard_user_id, shard)
>> ) WITH
>> bloom_filter_fp_chance=0.100000 AND
>> caching='KEYS_ONLY' AND
>> comment='' AND
>> dclocal_read_repair_chance=0.000000 AND
>> gc_grace_seconds=86400 AND
>> read_repair_chance=0.100000 AND
>> replicate_on_write='true' AND
>> populate_io_cache_on_flush='false' AND
>> compaction={'class': 'LeveledCompactionStrategy'} AND
>> compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
>>
>> If I then execute the following via CQL2, I do see the property. Seems odd to me?
>>
>> alter table shard_user_lookup with compaction_strategy_options:sstable_size_in_mb=256;
>>
>> CREATE TABLE shard_user_lookup (
>> shard_user_id int,
>> shard text,
>> creation_time timestamp,
>> status int,
>> user_id timeuuid,
>> PRIMARY KEY (shard_user_id, shard)
>> ) WITH
>> bloom_filter_fp_chance=0.100000 AND
>> caching='KEYS_ONLY' AND
>> comment='' AND
>> dclocal_read_repair_chance=0.000000 AND
>> gc_grace_seconds=86400 AND
>> read_repair_chance=0.100000 AND
>> replicate_on_write='true' AND
>> populate_io_cache_on_flush='false' AND
>> compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'} AND
>> compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
>>
>>
>> From: Wei Zhu <wz...@yahoo.com>
>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>, Wei Zhu <wz...@yahoo.com>
>> Date: Wednesday, July 24, 2013 8:49 PM
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Subject: Re: sstable size change
>>
>> what is output of show keyspaces from cassandra-cli, did you see the new value?
>>
>> Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
>> Compaction Strategy Options:
>> sstable_size_in_mb: XXX
>>
>> From: Keith Wright <kw...@nanigans.com>
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Sent: Wednesday, July 24, 2013 3:44 PM
>> Subject: Re: sstable size change
>>
>> Hi all,
>>
>> This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
>>
>> Thanks
>>
>> SSTable count: 4965
>> SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
>> Space used (live): 29062393142
>> Space used (total): 29140547702
>> Number of Keys (estimate): 195103104
>> Memtable Columns Count: 441483
>> Memtable Data Size: 205486218
>> Memtable Switch Count: 243
>> Read Count: 154226729
>>
>> -rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
>> -rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
>>
>>
>> From: sankalp kohli <ko...@gmail.com>
>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Date: Tuesday, July 23, 2013 3:04 PM
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Subject: Re: sstable size change
>>
>> "Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
>> Yes. Let it compact and increase in size.
>>
>>
>> On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com> wrote:
>>> On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com> wrote:
>>>> Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
>>>
>>> You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
>>>
>>> But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
>>>
>>> =Rob
>>
>>
>>
>
Re: sstable size change
Posted by Keith Wright <kw...@nanigans.com>.
Version 1.2.4.
Original alter that does nothing: alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
Correct alter: alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
Thanks
From: aaron morton <aa...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, July 29, 2013 4:02 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
Sounds like a good idea.
Please describe the version and query you first used.
Thanks for taking the time to update the thread.
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 27/07/2013, at 12:57 AM, Keith Wright <kw...@nanigans.com>> wrote:
FYI. It appears that the proper command in CQL3 is the following:
alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
From: Keith Wright <kw...@nanigans.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Thursday, July 25, 2013 9:14 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Wei Zhu <wz...@yahoo.com>>
Subject: Re: sstable size change
Unfortunately the table in question is a CQL3 table so cassandra-cli will not output its describe:
WARNING: CQL3 tables are intentionally omitted from 'describe' output.
See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
However, I did figure out that apparently I was not setting the change properly. I was using the following alter via CQL3:
alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
But when I did a describe, I did not see the change.
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
If I then execute the following via CQL2, I do see the property. Seems odd to me?
alter table shard_user_lookup with compaction_strategy_options:sstable_size_in_mb=256;
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
From: Wei Zhu <wz...@yahoo.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Wei Zhu <wz...@yahoo.com>>
Date: Wednesday, July 24, 2013 8:49 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
what is output of show keyspaces from cassandra-cli, did you see the new value?
Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
Compaction Strategy Options:
sstable_size_in_mb: XXX
________________________________
From: Keith Wright <kw...@nanigans.com>>
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Sent: Wednesday, July 24, 2013 3:44 PM
Subject: Re: sstable size change
Hi all,
This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
Thanks
SSTable count: 4965
SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
Space used (live): 29062393142
Space used (total): 29140547702
Number of Keys (estimate): 195103104
Memtable Columns Count: 441483
Memtable Data Size: 205486218
Memtable Switch Count: 243
Read Count: 154226729
-rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
From: sankalp kohli <ko...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, July 23, 2013 3:04 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
"Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
Yes. Let it compact and increase in size.
On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com>> wrote:
On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com>> wrote:
Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
=Rob
Re: sstable size change
Posted by aaron morton <aa...@thelastpickle.com>.
> I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
Sounds like a good idea.
Please describe the version and query you first used.
Thanks for taking the time to update the thread.
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 27/07/2013, at 12:57 AM, Keith Wright <kw...@nanigans.com> wrote:
> FYI. It appears that the proper command in CQL3 is the following:
>
> alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
>
> I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
>
> alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
>
>
> From: Keith Wright <kw...@nanigans.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Thursday, July 25, 2013 9:14 AM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>, Wei Zhu <wz...@yahoo.com>
> Subject: Re: sstable size change
>
> Unfortunately the table in question is a CQL3 table so cassandra-cli will not output its describe:
>
> WARNING: CQL3 tables are intentionally omitted from 'describe' output.
> See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
>
> However, I did figure out that apparently I was not setting the change properly. I was using the following alter via CQL3:
>
> alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
>
> But when I did a describe, I did not see the change.
>
> CREATE TABLE shard_user_lookup (
> shard_user_id int,
> shard text,
> creation_time timestamp,
> status int,
> user_id timeuuid,
> PRIMARY KEY (shard_user_id, shard)
> ) WITH
> bloom_filter_fp_chance=0.100000 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.000000 AND
> gc_grace_seconds=86400 AND
> read_repair_chance=0.100000 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> compaction={'class': 'LeveledCompactionStrategy'} AND
> compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
>
> If I then execute the following via CQL2, I do see the property. Seems odd to me?
>
> alter table shard_user_lookup with compaction_strategy_options:sstable_size_in_mb=256;
>
> CREATE TABLE shard_user_lookup (
> shard_user_id int,
> shard text,
> creation_time timestamp,
> status int,
> user_id timeuuid,
> PRIMARY KEY (shard_user_id, shard)
> ) WITH
> bloom_filter_fp_chance=0.100000 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.000000 AND
> gc_grace_seconds=86400 AND
> read_repair_chance=0.100000 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'} AND
> compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
>
>
> From: Wei Zhu <wz...@yahoo.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>, Wei Zhu <wz...@yahoo.com>
> Date: Wednesday, July 24, 2013 8:49 PM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: sstable size change
>
> what is output of show keyspaces from cassandra-cli, did you see the new value?
>
> Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
> Compaction Strategy Options:
> sstable_size_in_mb: XXX
>
> From: Keith Wright <kw...@nanigans.com>
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Sent: Wednesday, July 24, 2013 3:44 PM
> Subject: Re: sstable size change
>
> Hi all,
>
> This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
>
> Thanks
>
> SSTable count: 4965
> SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
> Space used (live): 29062393142
> Space used (total): 29140547702
> Number of Keys (estimate): 195103104
> Memtable Columns Count: 441483
> Memtable Data Size: 205486218
> Memtable Switch Count: 243
> Read Count: 154226729
>
> -rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
> -rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
>
>
> From: sankalp kohli <ko...@gmail.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Tuesday, July 23, 2013 3:04 PM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: sstable size change
>
> "Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
> Yes. Let it compact and increase in size.
>
>
> On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com> wrote:
>> On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com> wrote:
>>> Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
>>
>> You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
>>
>> But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
>>
>> =Rob
>
>
>
Re: sstable size change
Posted by Keith Wright <kw...@nanigans.com>.
FYI. It appears that the proper command in CQL3 is the following:
alter table cookie_user_lookup with compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'};
I still would have expected my original alter command to at least have thrown an error. I assume I should open a bug for this?
alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
From: Keith Wright <kw...@nanigans.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Thursday, July 25, 2013 9:14 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Wei Zhu <wz...@yahoo.com>>
Subject: Re: sstable size change
Unfortunately the table in question is a CQL3 table so cassandra-cli will not output its describe:
WARNING: CQL3 tables are intentionally omitted from 'describe' output.
See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
However, I did figure out that apparently I was not setting the change properly. I was using the following alter via CQL3:
alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
But when I did a describe, I did not see the change.
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
If I then execute the following via CQL2, I do see the property. Seems odd to me?
alter table shard_user_lookup with compaction_strategy_options:sstable_size_in_mb=256;
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
From: Wei Zhu <wz...@yahoo.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Wei Zhu <wz...@yahoo.com>>
Date: Wednesday, July 24, 2013 8:49 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
what is output of show keyspaces from cassandra-cli, did you see the new value?
Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
Compaction Strategy Options:
sstable_size_in_mb: XXX
________________________________
From: Keith Wright <kw...@nanigans.com>>
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Sent: Wednesday, July 24, 2013 3:44 PM
Subject: Re: sstable size change
Hi all,
This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
Thanks
SSTable count: 4965
SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
Space used (live): 29062393142
Space used (total): 29140547702
Number of Keys (estimate): 195103104
Memtable Columns Count: 441483
Memtable Data Size: 205486218
Memtable Switch Count: 243
Read Count: 154226729
-rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
From: sankalp kohli <ko...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, July 23, 2013 3:04 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
"Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
Yes. Let it compact and increase in size.
On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com>> wrote:
On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com>> wrote:
Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
=Rob
Re: sstable size change
Posted by Keith Wright <kw...@nanigans.com>.
Unfortunately the table in question is a CQL3 table so cassandra-cli will not output its describe:
WARNING: CQL3 tables are intentionally omitted from 'describe' output.
See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
However, I did figure out that apparently I was not setting the change properly. I was using the following alter via CQL3:
alter table shard_user_lookup with compaction_strategy_options = {'sstable_size_in_mb':256};
But when I did a describe, I did not see the change.
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
If I then execute the following via CQL2, I do see the property. Seems odd to me?
alter table shard_user_lookup with compaction_strategy_options:sstable_size_in_mb=256;
CREATE TABLE shard_user_lookup (
shard_user_id int,
shard text,
creation_time timestamp,
status int,
user_id timeuuid,
PRIMARY KEY (shard_user_id, shard)
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'sstable_size_in_mb': '256', 'class': 'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1', 'sstable_compression': 'LZ4Compressor'};
From: Wei Zhu <wz...@yahoo.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Wei Zhu <wz...@yahoo.com>>
Date: Wednesday, July 24, 2013 8:49 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
what is output of show keyspaces from cassandra-cli, did you see the new value?
Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
Compaction Strategy Options:
sstable_size_in_mb: XXX
________________________________
From: Keith Wright <kw...@nanigans.com>>
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Sent: Wednesday, July 24, 2013 3:44 PM
Subject: Re: sstable size change
Hi all,
This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
Thanks
SSTable count: 4965
SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
Space used (live): 29062393142
Space used (total): 29140547702
Number of Keys (estimate): 195103104
Memtable Columns Count: 441483
Memtable Data Size: 205486218
Memtable Switch Count: 243
Read Count: 154226729
-rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
From: sankalp kohli <ko...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, July 23, 2013 3:04 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
"Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
Yes. Let it compact and increase in size.
On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com>> wrote:
On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com>> wrote:
Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
=Rob
Re: sstable size change
Posted by Wei Zhu <wz...@yahoo.com>.
what is output of show keyspaces from cassandra-cli, did you see the new value?
Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
Compaction Strategy Options:
sstable_size_in_mb: XXX
________________________________
From: Keith Wright <kw...@nanigans.com>
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Sent: Wednesday, July 24, 2013 3:44 PM
Subject: Re: sstable size change
Hi all,
This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
Thanks
SSTable count: 4965
SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
Space used (live): 29062393142
Space used (total): 29140547702
Number of Keys (estimate): 195103104
Memtable Columns Count: 441483
Memtable Data Size: 205486218
Memtable Switch Count: 243
Read Count: 154226729
-rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
From: sankalp kohli <ko...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Tuesday, July 23, 2013 3:04 PM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: sstable size change
"Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
Yes. Let it compact and increase in size.
On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com> wrote:
On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com> wrote:
>
>Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
>
>
>You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
>
>
>But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
>
>
>=Rob
Re: sstable size change
Posted by Keith Wright <kw...@nanigans.com>.
Hi all,
This morning I increased the SSTable size for one of my LCS via an alter command and saw at least one compaction run (I did not trigger a compaction via nodetool nor upgrades stables nor removing the .json file). But so far my data sizes appear the same at the default 5 MB (see below for output of ls –Sal as well as relevant portion of cfstats). Is this expected? I was hoping to see at least one file at the new 256 MB size I set.
Thanks
SSTable count: 4965
SSTables in each level: [0, 10, 112/100, 1027/1000, 3816, 0, 0, 0]
Space used (live): 29062393142
Space used (total): 29140547702
Number of Keys (estimate): 195103104
Memtable Columns Count: 441483
Memtable Data Size: 205486218
Memtable Switch Count: 243
Read Count: 154226729
-rw-rw-r-- 1 cassandra cassandra 5247564 Jul 18 01:33 users-shard_user_lookup-ib-97153-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247454 Jul 23 02:59 users-shard_user_lookup-ib-109063-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247421 Jul 20 14:58 users-shard_user_lookup-ib-103127-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247415 Jul 17 13:56 users-shard_user_lookup-ib-95761-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247379 Jul 21 02:44 users-shard_user_lookup-ib-104718-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247346 Jul 21 21:54 users-shard_user_lookup-ib-106280-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247242 Jul 3 19:41 users-shard_user_lookup-ib-66049-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247235 Jul 21 02:44 users-shard_user_lookup-ib-104737-Data.db
-rw-rw-r-- 1 cassandra cassandra 5247233 Jul 20 14:58 users-shard_user_lookup-ib-103169-Data.db
From: sankalp kohli <ko...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, July 23, 2013 3:04 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
"Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered"
Yes. Let it compact and increase in size.
On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com>> wrote:
On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com>> wrote:
Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
You see, when two (or more!) SSTables love each other very much, they sometimes decide they want to compact together..
But seriously, "yes." If you force all existing SSTables to level 0, it is as if you just flushed them all. Level compaction then does a whole lot of compaction, using the active table size.
=Rob
Re: sstable size change
Posted by sankalp kohli <ko...@gmail.com>.
"Will Cassandra force any newly compacted files to my new setting as
compactions are naturally triggered"
Yes. Let it compact and increase in size.
On Tue, Jul 23, 2013 at 9:38 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com>wrote:
>
>> Can you elaborate on what you mean by "let it take its own course
>> organically"? Will Cassandra force any newly compacted files to my new
>> setting as compactions are naturally triggered?
>>
>
> You see, when two (or more!) SSTables love each other very much, they
> sometimes decide they want to compact together..
>
> But seriously, "yes." If you force all existing SSTables to level 0, it is
> as if you just flushed them all. Level compaction then does a whole lot of
> compaction, using the active table size.
>
> =Rob
>
Re: sstable size change
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 23, 2013 at 6:48 AM, Keith Wright <kw...@nanigans.com> wrote:
> Can you elaborate on what you mean by "let it take its own course
> organically"? Will Cassandra force any newly compacted files to my new
> setting as compactions are naturally triggered?
>
You see, when two (or more!) SSTables love each other very much, they
sometimes decide they want to compact together..
But seriously, "yes." If you force all existing SSTables to level 0, it is
as if you just flushed them all. Level compaction then does a whole lot of
compaction, using the active table size.
=Rob
Re: sstable size change
Posted by Keith Wright <kw...@nanigans.com>.
Can you elaborate on what you mean by "let it take its own course organically"? Will Cassandra force any newly compacted files to my new setting as compactions are naturally triggered?
From: sankalp kohli <ko...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, July 22, 2013 4:48 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: sstable size change
You can remove the json file and that will be treated as all sstables are now in L0. Since you have lot of data, the compaction will take a very long time. See the comment below directly from Cassandra code. If you chose to do this, you might want to increase the rate of compaction by usual means. If you are on spinning, then it might be a very big problem.
During the time of compaction, the read performance will be impacted.
Unless there is a very urgent need to change the sstable size, I would change the size and let it take it own course organically.
// LevelDB gives each level a score of how much data it contains vs its ideal amount, and
// compacts the level with the highest score. But this falls apart spectacularly once you
// get behind. Consider this set of levels:
// L0: 988 [ideal: 4]
// L1: 117 [ideal: 10]
// L2: 12 [ideal: 100]
//
// The problem is that L0 has a much higher score (almost 250) than L1 (11), so what we'll
// do is compact a batch of MAX_COMPACTING_L0 sstables with all 117 L1 sstables, and put the
// result (say, 120 sstables) in L1. Then we'll compact the next batch of MAX_COMPACTING_L0,
// and so forth. So we spend most of our i/o rewriting the L1 data with each batch.
//
// If we could just do *all* L0 a single time with L1, that would be ideal. But we can't
// -- see the javadoc for MAX_COMPACTING_L0.
//
// LevelDB's way around this is to simply block writes if L0 compaction falls behind.
// We don't have that luxury.
//
// So instead, we
// 1) force compacting higher levels first, which minimizes the i/o needed to compact
// optimially which gives us a long term win, and
// 2) if L0 falls behind, we will size-tiered compact it to reduce read overhead until
// we can catch up on the higher levels.
//
// This isn't a magic wand -- if you are consistently writing too fast for LCS to keep
// up, you're still screwed. But if instead you have intermittent bursts of activity,
// it can help a lot.
On Mon, Jul 22, 2013 at 12:51 PM, Andrew Bialecki <an...@gmail.com>> wrote:
My understanding is deleting the .json metadata file is the only way currently. If you search the user list archives, there are folks who are building tools to force compaction and rebuild sstables with the new size. I believe there's been a bit of talk of potentially including those tools as a pat of a future release.
Also, to answer your question about bloom filters, those are handled differently and if you run upgradesstables after altering the BF FP ratio, that will rebuild the BFs for each sstable.
On Mon, Jul 22, 2013 at 2:49 PM, Janne Jalkanen <ja...@ecyrd.com>> wrote:
I don't think upgradesstables is enough, since it's more of a "change this file to a new format but don't try to merge sstables and compact" -thing.
Deleting the .json -file is probably the only way, but someone more familiar with cassandra LCS might be able to tell whether manually editing the json file so that you drop all sstables a level might work? Since they would overflow the new level, they would compact soon, but the impact might be less drastic than just deleting the .json file (which takes everything to L0)...
/Janne
On 22 Jul 2013, at 16:02, Keith Wright <kw...@nanigans.com>> wrote:
Hi all,
I know there has been several threads recently on this but I wanted to make sure I got a clear answer: we are looking to increase our SSTable size for a couple of our LCS tables as well as chunk size (to match the SSD block size). The largest table is at 500 GB across 6 nodes (RF 3, C* 1.2.4 VNodes). I wanted to get feedback on the best way to make this change with minimal load impact on the cluster. After I make the change, I understand that I need to force the nodes to re-compact the tables.
Can this be done via upgrade sstables or do I need to shutdown the node, delete the .json file, and restart as some have suggested?
I assume I can do this one node at a time?
If I change the bloom filter size, I assume I will need to force compaction again? Using the same methodology?
Thank you
Re: sstable size change
Posted by sankalp kohli <ko...@gmail.com>.
You can remove the json file and that will be treated as all sstables are
now in L0. Since you have lot of data, the compaction will take a very long
time. See the comment below directly from Cassandra code. If you chose to
do this, you might want to increase the rate of compaction by usual means.
If you are on spinning, then it might be a very big problem.
During the time of compaction, the read performance will be impacted.
Unless there is a very urgent need to change the sstable size, I would
change the size and let it take it own course organically.
// LevelDB gives each level a score of how much data it contains vs its
ideal amount, and
// compacts the level with the highest score. But this falls apart
spectacularly once you
// get behind. Consider this set of levels:
// L0: 988 [ideal: 4]
// L1: 117 [ideal: 10]
// L2: 12 [ideal: 100]
//
// The problem is that L0 has a much higher score (almost 250) than
L1 (11), so what we'll
// do is compact a batch of MAX_COMPACTING_L0 sstables with all 117
L1 sstables, and put the
// result (say, 120 sstables) in L1. Then we'll compact the next
batch of MAX_COMPACTING_L0,
// and so forth. So we spend most of our i/o rewriting the L1 data
with each batch.
//
// If we could just do *all* L0 a single time with L1, that would
be ideal. But we can't
// -- see the javadoc for MAX_COMPACTING_L0.
//
// LevelDB's way around this is to simply block writes if L0
compaction falls behind.
// We don't have that luxury.
//
// So instead, we
// 1) force compacting higher levels first, which minimizes the i/o
needed to compact
// optimially which gives us a long term win, and
// 2) if L0 falls behind, we will size-tiered compact it to reduce
read overhead until
// we can catch up on the higher levels.
//
// This isn't a magic wand -- if you are consistently writing too
fast for LCS to keep
// up, you're still screwed. But if instead you have intermittent
bursts of activity,
// it can help a lot.
On Mon, Jul 22, 2013 at 12:51 PM, Andrew Bialecki <andrew.bialecki@gmail.com
> wrote:
> My understanding is deleting the .json metadata file is the only way
> currently. If you search the user list archives, there are folks who are
> building tools to force compaction and rebuild sstables with the new size.
> I believe there's been a bit of talk of potentially including those tools
> as a pat of a future release.
>
> Also, to answer your question about bloom filters, those are handled
> differently and if you run upgradesstables after altering the BF FP ratio,
> that will rebuild the BFs for each sstable.
>
>
> On Mon, Jul 22, 2013 at 2:49 PM, Janne Jalkanen <ja...@ecyrd.com>wrote:
>
>>
>> I don't think upgradesstables is enough, since it's more of a "change
>> this file to a new format but don't try to merge sstables and compact"
>> -thing.
>>
>> Deleting the .json -file is probably the only way, but someone more
>> familiar with cassandra LCS might be able to tell whether manually editing
>> the json file so that you drop all sstables a level might work? Since they
>> would overflow the new level, they would compact soon, but the impact might
>> be less drastic than just deleting the .json file (which takes everything
>> to L0)...
>>
>> /Janne
>>
>> On 22 Jul 2013, at 16:02, Keith Wright <kw...@nanigans.com> wrote:
>>
>> Hi all,
>>
>> I know there has been several threads recently on this but I wanted to
>> make sure I got a clear answer: we are looking to increase our SSTable
>> size for a couple of our LCS tables as well as chunk size (to match the SSD
>> block size). The largest table is at 500 GB across 6 nodes (RF 3, C*
>> 1.2.4 VNodes). I wanted to get feedback on the best way to make this
>> change with minimal load impact on the cluster. After I make the change, I
>> understand that I need to force the nodes to re-compact the tables.
>>
>> Can this be done via upgrade sstables or do I need to shutdown the node,
>> delete the .json file, and restart as some have suggested?
>>
>> I assume I can do this one node at a time?
>>
>> If I change the bloom filter size, I assume I will need to force
>> compaction again? Using the same methodology?
>>
>> Thank you
>>
>>
>>
>
Re: sstable size change
Posted by Andrew Bialecki <an...@gmail.com>.
My understanding is deleting the .json metadata file is the only way
currently. If you search the user list archives, there are folks who are
building tools to force compaction and rebuild sstables with the new size.
I believe there's been a bit of talk of potentially including those tools
as a pat of a future release.
Also, to answer your question about bloom filters, those are handled
differently and if you run upgradesstables after altering the BF FP ratio,
that will rebuild the BFs for each sstable.
On Mon, Jul 22, 2013 at 2:49 PM, Janne Jalkanen <ja...@ecyrd.com>wrote:
>
> I don't think upgradesstables is enough, since it's more of a "change this
> file to a new format but don't try to merge sstables and compact" -thing.
>
> Deleting the .json -file is probably the only way, but someone more
> familiar with cassandra LCS might be able to tell whether manually editing
> the json file so that you drop all sstables a level might work? Since they
> would overflow the new level, they would compact soon, but the impact might
> be less drastic than just deleting the .json file (which takes everything
> to L0)...
>
> /Janne
>
> On 22 Jul 2013, at 16:02, Keith Wright <kw...@nanigans.com> wrote:
>
> Hi all,
>
> I know there has been several threads recently on this but I wanted to
> make sure I got a clear answer: we are looking to increase our SSTable
> size for a couple of our LCS tables as well as chunk size (to match the SSD
> block size). The largest table is at 500 GB across 6 nodes (RF 3, C*
> 1.2.4 VNodes). I wanted to get feedback on the best way to make this
> change with minimal load impact on the cluster. After I make the change, I
> understand that I need to force the nodes to re-compact the tables.
>
> Can this be done via upgrade sstables or do I need to shutdown the node,
> delete the .json file, and restart as some have suggested?
>
> I assume I can do this one node at a time?
>
> If I change the bloom filter size, I assume I will need to force
> compaction again? Using the same methodology?
>
> Thank you
>
>
>
Re: sstable size change
Posted by Janne Jalkanen <ja...@ecyrd.com>.
I don't think upgradesstables is enough, since it's more of a "change this file to a new format but don't try to merge sstables and compact" -thing.
Deleting the .json -file is probably the only way, but someone more familiar with cassandra LCS might be able to tell whether manually editing the json file so that you drop all sstables a level might work? Since they would overflow the new level, they would compact soon, but the impact might be less drastic than just deleting the .json file (which takes everything to L0)...
/Janne
On 22 Jul 2013, at 16:02, Keith Wright <kw...@nanigans.com> wrote:
> Hi all,
>
> I know there has been several threads recently on this but I wanted to make sure I got a clear answer: we are looking to increase our SSTable size for a couple of our LCS tables as well as chunk size (to match the SSD block size). The largest table is at 500 GB across 6 nodes (RF 3, C* 1.2.4 VNodes). I wanted to get feedback on the best way to make this change with minimal load impact on the cluster. After I make the change, I understand that I need to force the nodes to re-compact the tables.
>
> Can this be done via upgrade sstables or do I need to shutdown the node, delete the .json file, and restart as some have suggested?
>
> I assume I can do this one node at a time?
>
> If I change the bloom filter size, I assume I will need to force compaction again? Using the same methodology?
>
> Thank you