You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Yuxing Yao (JIRA)" <ji...@apache.org> on 2016/11/28 08:20:58 UTC

[jira] [Created] (HIVE-15290) Stripe size smaller than specified.

Yuxing Yao created HIVE-15290:
---------------------------------

             Summary: Stripe size smaller than specified.
                 Key: HIVE-15290
                 URL: https://issues.apache.org/jira/browse/HIVE-15290
             Project: Hive
          Issue Type: Bug
          Components: ORC
    Affects Versions: 2.0.1, 2.1.0, 2.0.0, 1.2.1, 1.2.0
            Reporter: Yuxing Yao


In Hive-1.2.0, the real stripe size of output orc file will be very small if most of table data are empty, result in too many Column Statistics objects consumes most of the memory.
I found it become better in Hive-2.0.1, but the stripe size still much smaller than expected.
I saw there's a Jira item: https://issues.apache.org/jira/browse/HIVE-13232 moved the compressed = null out of if block, this changes helps a lot, but for completely fix this, another change is needed in `OutStream.getBufferSize()`
I've created the PR:
https://github.com/apache/hive/pull/118
Please take a look.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)