You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Cogumelos Maravilha <co...@sapo.pt> on 2017/07/01 09:50:15 UTC
Deflate compressor
Hi list,
Is there a way to set Deflate level of compression?
Brotli sounds good but unstable.
I just need more compression ratio.
I'm using C* 3.11.0
Cheers.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Deflate compressor
Posted by Cogumelos Maravilha <co...@sapo.pt>.
*"Decrease tombstone_compaction_interval and decrease
tombstone_threshold, or set unchecked_tombstone_compaction to true to
ignore both conditions and collect based /purely on gc grace/."*
Is this actually true for C* version 3.11.0?
AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '1', 'compaction_window_unit': 'HOURS',
'max_threshold': '128', 'min_threshold': '2',|*'unchecked_tombstone_compaction': 'true'*|}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.DeflateCompressor'}
Is this approach enough?
Thanks.
On 07/06/2017 06:27 PM, Jeff Jirsa wrote:
>
> On 2017-07-06 01:37 (-0700), Cogumelos Maravilha <co...@sapo.pt> wrote:
>> Hi Jeff,
>>
>> Thanks for your reply. But I've already changed from LZ4 to Deflate to
>> get higher compression level. Can I in the Deflate compressor do the
>> same, set for a higher level of compression?
>
> Not at this time; if it's important to you, please open a JIRA (as always, patches from the community are welcome)
>
>
>> Another question to the creator of the TimeWindowsCompactionStrategy:
>>
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
>> 'compaction_window_size': '1', 'compaction_window_unit': 'HOURS',
>> 'max_threshold': '128', 'min_threshold': '2'}
>> AND compression = {'chunk_length_in_kb': '64', 'class':
>> 'org.apache.cassandra.io.compress.DeflateCompressor'}
>>
>> There are some days that I have exactly 24 SSTables:
>> ls -alFh *Data*|grep 'Jul 3'|wc
>> 24
>> Others no:
>> ls -alFh *Data*|grep 'Jul 2'|wc
>> 59
>>
>> Is this normal?
> "Maybe", you could use sstablemetadata to get the maxTimestamp from the table, that's what TWCS will use to group data files together. Keep in mind that repaired sstables won't compact with unrepaired (if you're using incremental repair), and that tombstone compaction subproperties (which I don't see in your options, but maybe you had set before) can cause single sstable compactions that change the timestamp of the FILE, but the data within it may continue to be much older.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
Re: Deflate compressor
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-07-06 01:37 (-0700), Cogumelos Maravilha <co...@sapo.pt> wrote:
> Hi Jeff,
>
> Thanks for your reply. But I've already changed from LZ4 to Deflate to
> get higher compression level. Can I in the Deflate compressor do the
> same, set for a higher level of compression?
Not at this time; if it's important to you, please open a JIRA (as always, patches from the community are welcome)
>
> Another question to the creator of the TimeWindowsCompactionStrategy:
>
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '1', 'compaction_window_unit': 'HOURS',
> 'max_threshold': '128', 'min_threshold': '2'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.DeflateCompressor'}
>
> There are some days that I have exactly 24 SSTables:
> ls -alFh *Data*|grep 'Jul 3'|wc
> 24
> Others no:
> ls -alFh *Data*|grep 'Jul 2'|wc
> 59
>
> Is this normal?
"Maybe", you could use sstablemetadata to get the maxTimestamp from the table, that's what TWCS will use to group data files together. Keep in mind that repaired sstables won't compact with unrepaired (if you're using incremental repair), and that tombstone compaction subproperties (which I don't see in your options, but maybe you had set before) can cause single sstable compactions that change the timestamp of the FILE, but the data within it may continue to be much older.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Deflate compressor
Posted by Cogumelos Maravilha <co...@sapo.pt>.
Hi Jeff,
Thanks for your reply. But I've already changed from LZ4 to Deflate to
get higher compression level. Can I in the Deflate compressor do the
same, set for a higher level of compression?
Another question to the creator of the TimeWindowsCompactionStrategy:
AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '1', 'compaction_window_unit': 'HOURS',
'max_threshold': '128', 'min_threshold': '2'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.DeflateCompressor'}
There are some days that I have exactly 24 SSTables:
ls -alFh *Data*|grep 'Jul 3'|wc
24
Others no:
ls -alFh *Data*|grep 'Jul 2'|wc
59
Is this normal?
Thanks in advance.
On 06-07-2017 06:30, Jeff Jirsa wrote:
>
> On 2017-07-01 02:50 (-0700), Cogumelos Maravilha <co...@sapo.pt> wrote:
>> Hi list,
>>
>> Is there a way to set Deflate level of compression?
>>
>> Brotli sounds good but unstable.
>>
>> I just need more compression ratio.
>>
>> I'm using C* 3.11.0
>>
> LZ4 compression levels are configurable as of cassandra 3.6 ( https://issues.apache.org/jira/browse/CASSANDRA-11051 ) .
>
> "The attached patch adds the following optional parameters 'lz4_compressor_type' and 'lz4_high_compressor_level' to the LZ4Compressor."
>
> Glancing at the source: https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L41-L49
>
> lz4_high_compressor_level defaults to 9.
> lz4_compressor_type defaults to 'fast' ('high' is also available).
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Deflate compressor
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-07-01 02:50 (-0700), Cogumelos Maravilha <co...@sapo.pt> wrote:
> Hi list,
>
> Is there a way to set Deflate level of compression?
>
> Brotli sounds good but unstable.
>
> I just need more compression ratio.
>
> I'm using C* 3.11.0
>
LZ4 compression levels are configurable as of cassandra 3.6 ( https://issues.apache.org/jira/browse/CASSANDRA-11051 ) .
"The attached patch adds the following optional parameters 'lz4_compressor_type' and 'lz4_high_compressor_level' to the LZ4Compressor."
Glancing at the source: https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L41-L49
lz4_high_compressor_level defaults to 9.
lz4_compressor_type defaults to 'fast' ('high' is also available).
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org