[jira] [Commented] (IMPALA-14700) Add support for Parquet's LZ4_RAW compression

["Balazs Hevele (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062381#comment-18062381 ] 

Balazs Hevele commented on IMPALA-14700:
----------------------------------------

Patch at [https://gerrit.cloudera.org/#/c/24059/]
Note: This change requires a parquet version higher than we currently have (which is 1.12.3) for the LZ4_RAW enum value.

> Add support for Parquet's LZ4_RAW compression
> ---------------------------------------------
>
>                 Key: IMPALA-14700
>                 URL: https://issues.apache.org/jira/browse/IMPALA-14700
>             Project: IMPALA
>          Issue Type: Task
>          Components: Backend
>    Affects Versions: Impala 5.0.0
>            Reporter: Joe McDonnell
>            Assignee: Balazs Hevele
>            Priority: Major
>
> Parquet's current LZ4 compression uses a framing mechanism from Hadoop. Parquet decided to deprecate this and instead introduced the LZ4_RAW compression without the Hadoop framing. See https://issues.apache.org/jira/browse/PARQUET-1996 / https://issues.apache.org/jira/browse/PARQUET-2032
> We should add support for reading / writing LZ4_RAW. This should be fairly simple, as LZ4_RAW just uses the block compression directly. It should correspond to Lz4Compressor rather than Lz4BlockCompressor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org