You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Ryan Blue (JIRA)" <ji...@apache.org> on 2016/12/01 20:49:58 UTC
[jira] [Updated] (PARQUET-787) Add a size limit for heap
allocations when reading
[ https://issues.apache.org/jira/browse/PARQUET-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan Blue updated PARQUET-787:
------------------------------
Issue Type: Improvement (was: Bug)
> Add a size limit for heap allocations when reading
> --------------------------------------------------
>
> Key: PARQUET-787
> URL: https://issues.apache.org/jira/browse/PARQUET-787
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.9.0
> Reporter: Ryan Blue
> Assignee: Ryan Blue
>
> [G1GC allocates humongous objects directly in the old generation|https://www.infoq.com/articles/tuning-tips-G1-GC] to avoid unnecessary copies, which means that these allocations aren't garbage collected until a full GC runs. Humongous objects are objects that are 50% of the region size or more. Region size is at most 32MB (see the table for [region size from heap size|http://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection#Regions]).
> Parquet currently allocates a huge buffer for each contiguous group of column chunks, which in many cases is not garbage collected until a full GC. Adding a size limit for the allocation size should allow users to break row groups across multiple buffers so that buffers get collected when they have been read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)