[jira] [Commented] (HIVE-28979) ORC with ZSTD as default

["yongzhi.shao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-28979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17955401#comment-17955401 ] 

yongzhi.shao commented on HIVE-28979:
-------------------------------------

We used ZSTD compression during the use of HIVE - ICEBERG. Currently, we have found two problems:

1.Due to the high compression ratio of ZSTD, the algorithm in HIVE that originally estimates whether to enable MAP JOIN based on file size will be partially interfered. When using the ZSTD algorithm, HIVE often broadcasts larger data sets. Eventually, it causes OOM during the JOIN process.


2.The high compression ratio of ZSTD often affects the final number of reducers dynamically adjusted by TEZ. Compared with snappy, when reading zstd - compressed files, HIVE often dynamically adjusts to get fewer reducers, which seriously interferes with the running efficiency of SQL.

> ORC with ZSTD as default
> ------------------------
>
>                 Key: HIVE-28979
>                 URL: https://issues.apache.org/jira/browse/HIVE-28979
>             Project: Hive
>          Issue Type: Task
>            Reporter: Kokila N
>            Assignee: Kokila N
>            Priority: Major
>
> The ORC library has been upgraded to version *2.1.2* in Hive.
> Key changes in ORC 2.x:
>  * The *default compression codec* has changed from *ZLIB* to {*}ZSTD{*}.
> → [Related ORC PR: apache/orc#1733|https://github.com/apache/orc/pull/1733]
> However, in Hive, after upgrading from ORC *1.9.4* to {*}2.1.2{*}, we have explicitly retained *ZLIB* as the default compression codec, since we have not yet performed performance benchmarking to justify switching to ZSTD.
> Another notable change in ORC 2.x:
>  * *ZStandard (ZSTD)* compression has moved from using *aircompressor* to {*}zstd-jni{*}.
>  ** There is a fallback option to revert to {*}aircompressor{*}, which has been *enabled in Hive* for compatibility.
> → [Related ORC PR: apache/orc#988|https://github.com/apache/orc/pull/988]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)