[jira] [Resolved] (PARQUET-852) Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder

["Julien Le Dem (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Le Dem resolved PARQUET-852.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.10.0

Issue resolved by pull request 401
[https://github.com/apache/parquet-mr/pull/401]

> Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder
> ------------------------------------------------------------
>
>                 Key: PARQUET-852
>                 URL: https://issues.apache.org/jira/browse/PARQUET-852
>             Project: Parquet
>          Issue Type: Improvement
>            Reporter: John Jenkins
>            Priority: Minor
>             Fix For: 1.10.0
>
>
> The current allocation policy for ByteBasedBitPackingEncoder is to allocate 64KB * #bits up-front. As similarly observed in [PARQUET-580], this can lead to significant memory overheads for high-fanout scenarios (many columns and/or open files, in my case using BooleanPlainValuesWriter).
> As done in [PARQUET-585], I'll follow up with a PR that starts with a smaller buffer and works its way up to a max.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)