[jira] [Commented] (PARQUET-177) MemoryManager ensure minimum Column Chunk size

["Daniel Weeks (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312708#comment-14312708 ] 

Daniel Weeks commented on PARQUET-177:
--------------------------------------

Based on review comments, the limit is not based on row group size, but on an estimated minimum column chunk size.  This will help ensure that a "reasonable" amount of data will be written per column (default is page size).

> MemoryManager ensure minimum Column Chunk size
> ----------------------------------------------
>
>                 Key: PARQUET-177
>                 URL: https://issues.apache.org/jira/browse/PARQUET-177
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>    Affects Versions: 1.6.0rc2
>            Reporter: Daniel Weeks
>            Assignee: Daniel Weeks
>            Priority: Minor
>             Fix For: parquet-mr_1.6.0
>
>
> The memory manager currently has no limit to how small it will make row groups.  This is problematic because jobs that have a large number of writers can result in tiny row groups that hurt performance.
> The following patch will allow a configurable minimum size before killing the job.  Default is currently no limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)