[jira] [Commented] (ORC-409) Changes for extending MemoryManagerImpl

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634355#comment-16634355 ] 

ASF GitHub Bot commented on ORC-409:
------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/orc/pull/313


> Changes for extending MemoryManagerImpl
> ---------------------------------------
>
>                 Key: ORC-409
>                 URL: https://issues.apache.org/jira/browse/ORC-409
>             Project: ORC
>          Issue Type: Bug
>    Affects Versions: 1.5.4, 1.6.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>             Fix For: 1.5.4, 1.6.0
>
>
> Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50% (configurable) is available for orc writers. This works well if container model where container size is close to Xmx. In LLAP for example, a single container runs multiple executors and a single task take up 50% of heap which is typically much more that the task executors own memory limit. 
> Example: 128Gb container + 32 executors (assuming no cache) means there is 4GB available per executor. But a single task that opens multiple orc writers in the default case will assume 64GB memory is available for orc when in fact it should only use 2GB from memory per executor.
>  
> WriterOption provides a mechanism to plug-in a custom memory manager for such scenarios which is greate. But if the custom memory manager want to extend MemoryManagerImpl class, MemoryManagerImpl should use getTotalMemoryPool() everywhere so that custom memory manager can just Override getTotalMemoryPool() method and rest of the code from MemoryManagerImpl. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)