You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Phabricator (JIRA)" <ji...@apache.org> on 2013/05/01 01:18:16 UTC
[jira] [Updated] (HIVE-4421) Improve memory usage by ORC dictionaries

     [ https://issues.apache.org/jira/browse/HIVE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HIVE-4421:
------------------------------

    Attachment: HIVE-4421.D10545.2.patch

omalley updated the revision "HIVE-4421 [jira] Improve memory usage by ORC dictionaries".

  Changed the memory manager to check on each 5000 total rows added. This seems to give the best trade off between handling too many writers in a small heap and still managing memory pretty accurately.

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D10545

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D10545?vs=32889&id=33201#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicIntArray.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/MemoryManager.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OutStream.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/PositionedOutputStream.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RedBlackTree.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringRedBlackTree.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestMemoryManager.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestStringRedBlackTree.java
  ql/src/test/resources/orc-file-dump.out

To: JIRA, omalley

                
> Improve memory usage by ORC dictionaries
> ----------------------------------------
>
>                 Key: HIVE-4421
>                 URL: https://issues.apache.org/jira/browse/HIVE-4421
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.11.0
>
>         Attachments: HIVE-4421.D10545.1.patch, HIVE-4421.D10545.2.patch
>
>
> Currently, for tables with many string columns, it is possible to significantly underestimate the memory used by the ORC dictionaries and cause the query to run out of memory in the task. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira