[jira] [Commented] (ORC-409) Changes for extending MemoryManagerImpl

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631233#comment-16631233 ] 

ASF GitHub Bot commented on ORC-409:
------------------------------------

GitHub user prasanthj opened a pull request:

    https://github.com/apache/orc/pull/313

    ORC-409: Changes for extending MemoryManagerImpl

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/prasanthj/orc ORC-409

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/orc/pull/313.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #313
    
----
commit 9cc833d560da188fd48ece42c31d1daa9ea6bc08
Author: Prasanth Jayachandran <pr...@...>
Date:   2018-09-28T00:51:57Z

    ORC-409: Changes for extending MemoryManagerImpl

----


> Changes for extending MemoryManagerImpl
> ---------------------------------------
>
>                 Key: ORC-409
>                 URL: https://issues.apache.org/jira/browse/ORC-409
>             Project: ORC
>          Issue Type: Bug
>    Affects Versions: 1.5.4, 1.6.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>
> Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50% (configurable) is available for orc writers. This works well if container model where container size is close to Xmx. In LLAP for example, a single container runs multiple executors and a single task take up 50% of heap which is typically much more that the task executors own memory limit. 
> Example: 128Gb container + 32 executors (assuming no cache) means there is 4GB available per executor. But a single task that opens multiple orc writers in the default case will assume 64GB memory is available for orc when in fact it should only use 2GB from memory per executor.
>  
> WriterOption provides a mechanism to plug-in a custom memory manager for such scenarios which is greate. But if the custom memory manager want to extend MemoryManagerImpl class, MemoryManagerImpl should use getTotalMemoryPool() everywhere so that custom memory manager can just Override getTotalMemoryPool() method and rest of the code from MemoryManagerImpl. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)