[jira] [Created] (HBASE-26353) Support loadable dictionaries in hbase-compression-zstd

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

Andrew Kyle Purtell created HBASE-26353:
-------------------------------------------

             Summary: Support loadable dictionaries in hbase-compression-zstd
                 Key: HBASE-26353
                 URL: https://issues.apache.org/jira/browse/HBASE-26353
             Project: HBase
          Issue Type: Sub-task
            Reporter: Andrew Kyle Purtell
            Assignee: Andrew Kyle Purtell
             Fix For: 2.5.0, 3.0.0-alpha-2


ZStandard supports initialization of compressors and decompressors with a precomputed dictionary, which can dramatically improve and speed up compression of tables with small values. For more details, please see [The Case For Small Data Compression|https://github.com/facebook/zstd#the-case-for-small-data-compression]. 

If a table is going to have a lot of small values and the user can put together a representative set of files that can be used to train a dictionary for compressing those values, a dictionary can be trained with the {{zstd}} command line utility, available in any zstandard package for your favorite OS:

Training:
{noformat}
$ zstd --maxdict=1126400 --train-fastcover=shrink \
    -o mytable.dict training_files/*
Trying 82 different sets of parameters
...
k=674                                      
d=8
f=20
steps=40
split=75
accel=1
Save dictionary of size 1126400 into file mytable.dict
{noformat}

Deploy the dictionary file to HDFS.

Create the table:

{noformat}
hbase> create "mytable", 
  ... ,
  CONFIGURATION => {
    'hbase.io.compress.zstd.level' => '6',
    'hbase.io.compress.zstd.dictionary' => true,
    'hbase.io.compress.zstd.dictonary.file' => \
      'hdfs://nn/zdicts/mytable.dict'
  }
{noformat}

Now start storing data. Compression results even for small values will be excellent.

Note: Beware, if the dictionary is lost, the data will not be decompressable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)