You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "John Jenkins (JIRA)" <ji...@apache.org> on 2017/02/01 14:02:51 UTC

[jira] [Created] (PARQUET-852) Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder

John Jenkins created PARQUET-852:
------------------------------------

             Summary: Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder
                 Key: PARQUET-852
                 URL: https://issues.apache.org/jira/browse/PARQUET-852
             Project: Parquet
          Issue Type: Improvement
            Reporter: John Jenkins
            Priority: Minor


The current allocation policy for ByteBasedBitPackingEncoder is to allocate 64KB * #bits up-front. As similarly observed in [PARQUET-580], this can lead to significant memory overheads for high-fanout scenarios (many columns and/or open files, in my case using BooleanPlainValuesWriter).

As done in [PARQUET-585], I'll follow up with a PR that starts with a smaller buffer and works its way up to a max.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)