You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Daniel Weeks (JIRA)" <ji...@apache.org> on 2015/02/04 01:56:34 UTC

[jira] [Created] (PARQUET-177) MemoryManager ensure minimum Row Group size

Daniel Weeks created PARQUET-177:
------------------------------------

             Summary: MemoryManager ensure minimum Row Group size
                 Key: PARQUET-177
                 URL: https://issues.apache.org/jira/browse/PARQUET-177
             Project: Parquet
          Issue Type: Improvement
          Components: parquet-mr
    Affects Versions: 1.6.0rc2
            Reporter: Daniel Weeks
            Assignee: Daniel Weeks
            Priority: Minor


The memory manager currently has no limit to how small it will make row groups.  This is problematic because jobs that have a large number of writers can result in tiny row groups that hurt performance.

The following patch will allow a configurable minimum size before killing the job.  Default is currently no limit.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)