You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Zhiyuan Yang (JIRA)" <ji...@apache.org> on 2017/03/03 00:31:45 UTC
[jira] [Comment Edited] (TEZ-3647) Lower default Xmx for small AM
container
[ https://issues.apache.org/jira/browse/TEZ-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893377#comment-15893377 ]
Zhiyuan Yang edited comment on TEZ-3647 at 3/3/17 12:31 AM:
------------------------------------------------------------
This patch let Tez auto determine AM container size. The policy is: if AM container is large enough, we use 0.8 as default max heap fraction; otherwise we use 0.7. Both slab boundary and default fractions are configurable. Please help review [~sseth], [~rajesh.balamohan].
was (Author: aplusplus):
This patch let Tez auto determine AM container size. The policy is like slab allocation: if AM container is large enough, we use 0.8 as default max heap fraction; otherwise we use 0.7. Both slab boundary and default fractions are configurable. Please help review [~sseth], [~rajesh.balamohan].
> Lower default Xmx for small AM container
> -----------------------------------------
>
> Key: TEZ-3647
> URL: https://issues.apache.org/jira/browse/TEZ-3647
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Zhiyuan Yang
> Assignee: Zhiyuan Yang
> Attachments: TEZ-3647.1.patch
>
>
> The default Xmx is 80% of container size. For a 2G container, non-heap memory can cost more than 20%. Total off-heap memory is around 450MB. In case of nearly full heap memory usage, we will be out of physical memory and killed by YARN.
> Here is native memory summary for a AM running tpcds query 13.
> {code}
> Native Memory Tracking:
> Total: reserved=3424708KB, committed=2155712KB
> - Java Heap (reserved=1677312KB, committed=1677312KB)
> (mmap: reserved=1677312KB, committed=1677312KB)
> - Class (reserved=1096701KB, committed=54061KB)
> (classes #8465)
> (malloc=1021KB #11433)
> (mmap: reserved=1095680KB, committed=53040KB)
> - Thread (reserved=195103KB, committed=195103KB)
> (thread #191)
> (stack: reserved=193948KB, committed=193948KB)
> (malloc=611KB #986)
> (arena=543KB #382)
> - Code (reserved=254302KB, committed=31222KB)
> (malloc=4702KB #7076)
> (mmap: reserved=249600KB, committed=26520KB)
> - GC (reserved=148963KB, committed=148963KB)
> (malloc=53951KB #17562)
> (mmap: reserved=95012KB, committed=95012KB)
> - Compiler (reserved=306KB, committed=306KB)
> (malloc=176KB #311)
> (arena=131KB #3)
> - Internal (reserved=17872KB, committed=17872KB)
> (malloc=17840KB #15275)
> (mmap: reserved=32KB, committed=32KB)
> - Symbol (reserved=12659KB, committed=12659KB)
> (malloc=10829KB #106221)
> (arena=1830KB #1)
> - Native Memory Tracking (reserved=2735KB, committed=2735KB)
> (malloc=192KB #2968)
> (tracking overhead=2543KB)
> - Arena Chunk (reserved=5034KB, committed=5034KB)
> (malloc=5034KB)
> - Tracing (reserved=10444KB, committed=10444KB)
> (malloc=10444KB #274)
> - Unknown (reserved=3276KB, committed=0KB)
> (mmap: reserved=3276KB, committed=0KB)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)