You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Deepak Majeti (JIRA)" <ji...@apache.org> on 2016/09/16 19:40:20 UTC
[jira] [Commented] (PARQUET-684) [C++] Hardware optimizations for
dictionary / RLE encoding/decoding
[ https://issues.apache.org/jira/browse/PARQUET-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497185#comment-15497185 ]
Deepak Majeti commented on PARQUET-684:
---------------------------------------
I looked at the code and the blog briefly. The current implementation works for dictionary indices that are bit-packed.
This implementation will have to be extended to support Rle-Bitpacked hybrid encoding current used by parquet-cpp to encode dictionary index values.
Encoding details here: https://github.com/apache/parquet-cpp/blob/master/src/parquet/util/rle-encoding.h#L33
I guess the rle encoding of indices will furture improve the performance since it will not require the costly gather instruction.
> [C++] Hardware optimizations for dictionary / RLE encoding/decoding
> -------------------------------------------------------------------
>
> Key: PARQUET-684
> URL: https://issues.apache.org/jira/browse/PARQUET-684
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-cpp
> Reporter: Wes McKinney
> Assignee: Deepak Majeti
>
> See discussion in
> https://github.com/apache/parquet-cpp/pull/140
> and experiments from Daniel Lemire in
> https://github.com/lemire/dictionary
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)