[jira] [Commented] (HBASE-26353) Support loadable dictionaries in hbase-compression-zstd

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428482#comment-17428482 ] 

Andrew Kyle Purtell commented on HBASE-26353:
---------------------------------------------

Let me re-run the [performance evaluation from HBASE-26259|https://issues.apache.org/jira/browse/HBASE-26259?focusedCommentId=17422934&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17422934] but with synthetic small value data and compare speed and efficiency with precomputed dictionary vs without. Gains are expected but I'd like to present some hard comparison data here.

> Support loadable dictionaries in hbase-compression-zstd
> -------------------------------------------------------
>
>                 Key: HBASE-26353
>                 URL: https://issues.apache.org/jira/browse/HBASE-26353
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Andrew Kyle Purtell
>            Assignee: Andrew Kyle Purtell
>            Priority: Minor
>             Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> ZStandard supports initialization of compressors and decompressors with a precomputed dictionary, which can dramatically improve and speed up compression of tables with small values. For more details, please see [The Case For Small Data Compression|https://github.com/facebook/zstd#the-case-for-small-data-compression]. 
> If a table is going to have a lot of small values and the user can put together a representative set of files that can be used to train a dictionary for compressing those values, a dictionary can be trained with the {{zstd}} command line utility, available in any zstandard package for your favorite OS:
> Training:
> {noformat}
> $ zstd --maxdict=1126400 --train-fastcover=shrink \
>     -o mytable.dict training_files/*
> Trying 82 different sets of parameters
> ...
> k=674                                      
> d=8
> f=20
> steps=40
> split=75
> accel=1
> Save dictionary of size 1126400 into file mytable.dict
> {noformat}
> Deploy the dictionary file to HDFS or S3, etc.
> Create the table:
> {noformat}
> hbase> create "mytable", 
>   ... ,
>   CONFIGURATION => {
>     'hbase.io.compress.zstd.level' => '6',
>     'hbase.io.compress.zstd.dictionary' => true,
>     'hbase.io.compress.zstd.dictonary.file' =>  'hdfs://nn/zdicts/mytable.dict'
>   }
> {noformat}
> Now start storing data. Compression results even for small values will be excellent.
> Note: Beware, if the dictionary is lost, the data will not be decompressable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)