You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2017/03/03 21:45:45 UTC
[jira] [Updated] (TEZ-3647) Add a setting which lets Tez determine
Xmx
[ https://issues.apache.org/jira/browse/TEZ-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated TEZ-3647:
--------------------------------
Summary: Add a setting which lets Tez determine Xmx (was: Lower default Xmx for small AM container )
> Add a setting which lets Tez determine Xmx
> ------------------------------------------
>
> Key: TEZ-3647
> URL: https://issues.apache.org/jira/browse/TEZ-3647
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Zhiyuan Yang
> Assignee: Zhiyuan Yang
> Fix For: 0.9.0
>
> Attachments: TEZ-3647.1.patch, TEZ-3647.2.patch, TEZ-3647.3.patch
>
>
> The default Xmx is 80% of container size. For a 2G container, non-heap memory can cost more than 20%. Total off-heap memory is around 450MB. In case of nearly full heap memory usage, we will be out of physical memory and killed by YARN.
> Here is native memory summary for a AM running tpcds query 13.
> {code}
> Native Memory Tracking:
> Total: reserved=3424708KB, committed=2155712KB
> - Java Heap (reserved=1677312KB, committed=1677312KB)
> (mmap: reserved=1677312KB, committed=1677312KB)
> - Class (reserved=1096701KB, committed=54061KB)
> (classes #8465)
> (malloc=1021KB #11433)
> (mmap: reserved=1095680KB, committed=53040KB)
> - Thread (reserved=195103KB, committed=195103KB)
> (thread #191)
> (stack: reserved=193948KB, committed=193948KB)
> (malloc=611KB #986)
> (arena=543KB #382)
> - Code (reserved=254302KB, committed=31222KB)
> (malloc=4702KB #7076)
> (mmap: reserved=249600KB, committed=26520KB)
> - GC (reserved=148963KB, committed=148963KB)
> (malloc=53951KB #17562)
> (mmap: reserved=95012KB, committed=95012KB)
> - Compiler (reserved=306KB, committed=306KB)
> (malloc=176KB #311)
> (arena=131KB #3)
> - Internal (reserved=17872KB, committed=17872KB)
> (malloc=17840KB #15275)
> (mmap: reserved=32KB, committed=32KB)
> - Symbol (reserved=12659KB, committed=12659KB)
> (malloc=10829KB #106221)
> (arena=1830KB #1)
> - Native Memory Tracking (reserved=2735KB, committed=2735KB)
> (malloc=192KB #2968)
> (tracking overhead=2543KB)
> - Arena Chunk (reserved=5034KB, committed=5034KB)
> (malloc=5034KB)
> - Tracing (reserved=10444KB, committed=10444KB)
> (malloc=10444KB #274)
> - Unknown (reserved=3276KB, committed=0KB)
> (mmap: reserved=3276KB, committed=0KB)
> {code}
> Please notice the memory here means virtual memory size. In worst scenario all pages get loaded and takes same amount of physical memory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)