[jira] [Comment Edited] (HBASE-27323) Support to take the initiative to compact cold large files and compression diff hfiles after changing storefile compression algrithm

["Xiaolin Ha (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-27323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584572#comment-17584572 ] 

Xiaolin Ha edited comment on HBASE-27323 at 8/25/22 3:20 AM:
-------------------------------------------------------------

Yes, they are definitely different things.

The attribute "COMPRESSION_COMPACT, COMPRESSION_COMPACT_MAJOR, COMPRESSION_COMPACT_MINOR" aims to support different compression algorithms for one column family from the COMPRESSION type. I think it can use different compressions for recently flushed and colder data, since in some circumstances hot data are more recently and can use faster algorithms, while colder can use higher ratio algorithms. 

In this issue, it supports to quickly apply the changed compression algorithm to ALL the store files through the regular MINOR compactions, though there are some cold large files that rare participate in minor compactions. When we only change the "COMPRESSION" attribute, all the different compression algorithm store files whatever is large or small, should be rewrite to the new compression algorithm.


was (Author: xiaolin ha):
Yes, they are definitely different things.

The attribute "COMPRESSION_COMPACT, COMPRESSION_COMPACT_MAJOR, COMPRESSION_COMPACT_MINOR" aims to support different compression algorithms for one column family from the COMPRESSION type. I think it can use different compressions for recently flushed and colder data, since in some circumstances hot data are more recently and can used faster algorithms, while colder can use higher ratio algorithms. 

In this issue, it can support to quickly apply the changed compression algorithm to ALL the store files through the regular MINOR compactions, though there are some cold large files that rare participate in minor compactions. When we only change the "COMPRESSION" attribute, all the different compression algorithm store files whatever is large or small, should be rewrite to the new compression algorithm.

> Support to take the initiative to compact cold large files and compression diff hfiles after changing storefile compression algrithm 
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-27323
>                 URL: https://issues.apache.org/jira/browse/HBASE-27323
>             Project: HBase
>          Issue Type: Improvement
>          Components: Compaction
>    Affects Versions: 3.0.0-alpha-3, 2.4.13
>            Reporter: Xiaolin Ha
>            Priority: Major
>             Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> We can set switch to enable this feature, to make the compression algrithm changes be applied to all the existing store files, especially when we make a lower space usage compression algrithm change, e.g. from LZO to ZSTD can save more than 30% spaces.
> All the compaction policies should support this feature.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)