You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Samarth Jain (Jira)" <ji...@apache.org> on 2019/08/28 18:22:00 UTC

[jira] [Created] (PARQUET-1643) Use airlift non-native implementations for GZIP, LZ0 and LZ4 codecs

Samarth Jain created PARQUET-1643:
-------------------------------------

             Summary: Use airlift non-native implementations for GZIP, LZ0 and LZ4 codecs
                 Key: PARQUET-1643
                 URL: https://issues.apache.org/jira/browse/PARQUET-1643
             Project: Parquet
          Issue Type: Improvement
            Reporter: Samarth Jain


[~rdblue] pointed me to [https://github.com/airlift/aircompressor] which provides non-native implementations of compression codecs. It claims to be much faster than native wrappers that parquet uses. This Jira is to track the work needed for exploring using these codecs, getting benchmark results and making changes including not needing to pool compressors and decompressors anymore. Note that this doesn't include SNAPPY since Parquet already has its own non-hadoopy implementation for it. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)