You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Phabricator (JIRA)" <ji...@apache.org> on 2013/04/25 08:16:25 UTC

[jira] [Updated] (HIVE-4421) Improve memory usage by ORC dictionaries

     [ https://issues.apache.org/jira/browse/HIVE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phabricator updated HIVE-4421:
------------------------------

    Attachment: HIVE-4421.D10545.1.patch

omalley requested code review of "HIVE-4421 [jira] Improve memory usage by ORC dictionaries".

Reviewers: JIRA

HIVE-4421 Improve ORC dictionary memory usage and tracking

Currently, for tables with many string columns, it is possible to significantly underestimate the memory used by the ORC dictionaries and cause the query to run out of memory in the task.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D10545

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicIntArray.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RedBlackTree.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringRedBlackTree.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestStringRedBlackTree.java
  ql/src/test/resources/orc-file-dump.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/25221/

To: JIRA, omalley

                
> Improve memory usage by ORC dictionaries
> ----------------------------------------
>
>                 Key: HIVE-4421
>                 URL: https://issues.apache.org/jira/browse/HIVE-4421
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: HIVE-4421.D10545.1.patch
>
>
> Currently, for tables with many string columns, it is possible to significantly underestimate the memory used by the ORC dictionaries and cause the query to run out of memory in the task. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira