[jira] [Commented] (HIVE-11807) Set ORC buffer size in relation to set stripe size

["Owen O'Malley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744025#comment-14744025 ] 

Owen O'Malley commented on HIVE-11807:
--------------------------------------

Ok, there are a couple changes that I'd propose:
* Use the stripe size rather than the available memory. This is more important because the stripe will be flushed when the buffering reaches the stripe size.
* Count all of the columns not just the top level ones.
* Most of the streams have at most 2 large streams so if we use 20 buffers, that will give us a reasonable balance between internal fragmentation and throughput.


> Set ORC buffer size in relation to set stripe size
> --------------------------------------------------
>
>                 Key: HIVE-11807
>                 URL: https://issues.apache.org/jira/browse/HIVE-11807
>             Project: Hive
>          Issue Type: Improvement
>          Components: File Formats
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>
> A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of the streams only get a buffer or two before the stripe size is reached. The current code uses the available memory instead of the stripe size and thus doesn't shrink the buffer size if the JVM has much more memory than the stripe size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)