[jira] [Comment Edited] (PARQUET-678) Allow for custom compression codecs

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860404#comment-15860404 ] 

Wes McKinney edited comment on PARQUET-678 at 2/9/17 11:39 PM:
---------------------------------------------------------------

The format also provides for Brotli compression: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L331

I am sure that LZ4 and zstd would be welcome additions -- at least on the C++ side adding these would not cause us much hardship (we have added Brotli support already)


was (Author: wesmckinn):
The format also provides for Brotli compression: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L331

I am sure that LZ4 and zstd would be welcome additions -- at least on the C++ side adding these would not cause us much hardshop (we have added Brotli support already)

> Allow for custom compression codecs
> -----------------------------------
>
>                 Key: PARQUET-678
>                 URL: https://issues.apache.org/jira/browse/PARQUET-678
>             Project: Parquet
>          Issue Type: Wish
>            Reporter: Steven Anton
>            Priority: Minor
>
> I understand that the list of accepted compression codecs is explicity limited to uncompressed, snappy, gzip, and lzo. (See parquet.hadoop.metadata.CompressionCodecName.java) Is there a reason for this? Or is there an easy workaround? On the surface it seems like an unnecessary restriction.
> I ask because I have written a custom codec to implement encryption and I'm unable to use it with Parquet, which is a real shame because it is the main storage format I was hoping to use.
> Other thoughts on how to implement encryption in Parquet with this limitation?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)