[jira] [Commented] (HIVE-10466) LLAP: fix container sizing configuration for memory

["Gopal V (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509794#comment-14509794 ] 

Gopal V commented on HIVE-10466:
--------------------------------

The current memory script does this - it uses NODE_MEM/2 as the per-instance-executor & uses 1Gb by default as the cache.

Then it goes through a bunch of complex "heuristics" to produce a complete configuration listing which contains the YARN container.size, the Xmx and the total memory allocated to executors.
 
https://github.com/apache/hive/blob/llap/llap-server/src/main/resources/package.py#L14

This produces a workable configuration, but it misses the total capacity of the node by a significant margin (will be >60%, so no double allocs on a single node, but will be < 100%).

Even in that script, the yarn min-alloc is missing from the computation, so as we edge closer to the line, the harder it gets to configure this correctly.

After that, there's the whole YARN reserved memory fraction to deal with in this, so that we can avoid taking up memory in YARN that LLAP can't use.

> LLAP: fix container sizing configuration for memory
> ---------------------------------------------------
>
>                 Key: HIVE-10466
>                 URL: https://issues.apache.org/jira/browse/HIVE-10466
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Gopal V
>            Assignee: Vikram Dixit K
>
> We cannot use full machine for LLAP due to config for cache and executors being "split brain"... please refer to  [~gopalv] for details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)