You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Kevin Wilfong (JIRA)" <ji...@apache.org> on 2013/03/13 00:11:13 UTC

[jira] [Created] (HIVE-4157) ORC runs out of heap when writing

Kevin Wilfong created HIVE-4157:
-----------------------------------

             Summary: ORC runs out of heap when writing
                 Key: HIVE-4157
                 URL: https://issues.apache.org/jira/browse/HIVE-4157
             Project: Hive
          Issue Type: Improvement
          Components: Serializers/Deserializers
    Affects Versions: 0.11.0
            Reporter: Kevin Wilfong
            Assignee: Kevin Wilfong


The OutStream class used by the ORC file format seems to aggressively allocate memory for ByteBuffers and doesn't seem too eager to give it back.

This causes issues with heap space, particularly when a wide tables/dynamic partitions are involved.

As a first step to resolving this problem, the OutStream class can be modified to lazily allocate memory, and more actively make it available for garbage collection.

Follow ups could include checking the amount of free memory as part of determining if a spill is needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira