You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Andrew Sherman (JIRA)" <ji...@apache.org> on 2018/09/14 23:34:00 UTC
[jira] [Updated] (IMPALA-6658) Parquet RLE encoding can waste space
with small repeated runs
[ https://issues.apache.org/jira/browse/IMPALA-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Sherman updated IMPALA-6658:
-----------------------------------
Description:
Currently RleEncoder creates repeated runs from 8 repeated values, which can be less space efficient than bit-packed if bit width is 1 or 2. In the worst case, the whole data page can be ~2X larger if bit width is 1, and ~1.25X larger if bit is 2 compared to bit-packing.
A comment in rle_encoding.h writes different numbers, but it probably does not calculate with the overhead of splitting long runs into smaller ones (every run adds +1 byte for its length): [https://github.com/apache/impala/blob/8079cd9d2a87051f81a41910b74fab15e35f36ea/be/src/util/rle-encoding.h#L62]
Note that if the data page is compressed, this size difference probably disappears, but the larger uncompressed buffer size can still affect performance.
Parquet RLE encoding is described here:
[https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding-bit-packing-hybrid-rle-3]
was:
Currently RleEncoder creates repeated runs from 8 repeated values, which can be less space efficient than bit-packed if bit width is 1 or 2. In the worst case, the whole data page can be ~2X larger if bit width is 1, and ~1.25X larger if bit is 2 compared to bit-packing.
A comment in rle_encoding.h writes different numbers, but it probably does not calculate with the overhead of splitting long runs into smaller ones (every run adds +1 byte for its length): https://github.com/apache/impala/blob/8079cd9d2a87051f81a41910b74fab15e35f36ea/be/src/util/rle-encoding.h#L62
Note that if the data page is compressed, this size difference probably disappears, but the larger uncompressed buffer size can still affect performance.
Parquet RLE encoding is described here: https://github.com/apache/parquet-
format/blob/master/Encodings.md#run-length-encoding--bit-packing-hybrid-rle--3
> Parquet RLE encoding can waste space with small repeated runs
> -------------------------------------------------------------
>
> Key: IMPALA-6658
> URL: https://issues.apache.org/jira/browse/IMPALA-6658
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Csaba Ringhofer
> Assignee: Andrew Sherman
> Priority: Minor
> Labels: parquet, ramp-up
>
> Currently RleEncoder creates repeated runs from 8 repeated values, which can be less space efficient than bit-packed if bit width is 1 or 2. In the worst case, the whole data page can be ~2X larger if bit width is 1, and ~1.25X larger if bit is 2 compared to bit-packing.
> A comment in rle_encoding.h writes different numbers, but it probably does not calculate with the overhead of splitting long runs into smaller ones (every run adds +1 byte for its length): [https://github.com/apache/impala/blob/8079cd9d2a87051f81a41910b74fab15e35f36ea/be/src/util/rle-encoding.h#L62]
> Note that if the data page is compressed, this size difference probably disappears, but the larger uncompressed buffer size can still affect performance.
> Parquet RLE encoding is described here:
> [https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding-bit-packing-hybrid-rle-3]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org