[jira] [Updated] (GIRAPH-812) Make heap use configurable fraction of total container memory

["Alexandre Fonseca (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GIRAPH-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexandre Fonseca updated GIRAPH-812:
-------------------------------------

    Attachment: GIRAPH-812.patch

The attached patch modifies the properties surrounding yarn memory allocation for the ApplicationMaster and tasks. For the ApplicationMaster, total memory allocation is hardcoded at 512MB and heap usage at 75% of that. For the tasks, a default of 1024MB and heap usage of 75% is used. These defaults can be overriden by the following properties/command line options:
* giraph.yarn.task.mem.mb / -ym 1024
* giraph.yarn.task.heap.fraction / -yh 0.75

The patch passes mvn verify and has been tested with the SimpleShortestPaths example in a local yarn installation.

> Make heap use configurable fraction of total container memory
> -------------------------------------------------------------
>
>                 Key: GIRAPH-812
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-812
>             Project: Giraph
>          Issue Type: Improvement
>          Components: conf and scripts
>    Affects Versions: 1.1.0
>            Reporter: Alexandre Fonseca
>            Priority: Minor
>              Labels: yarn
>         Attachments: GIRAPH-812.patch
>
>
> Currently, both the Yarn ApplicationMaster and task containers are launched with a JVM heap size matching the total memory available to the container. However, while the heap is indeed one of the major users of memory in the JVM, other components also occupy memory which is not included in the heap calculation: http://stackoverflow.com/questions/9725633/why-is-my-jvms-total-memory-usage-more-than-30-times-greater-than-its-xmx-value. 
> With current code, if we attempt to launch containers with limited memory
> (e.g 512MB) and leave the default yarn.nodemanager.vmem-pmem-ratio of 2.1, it becomes very easy to overflow the maximum amount of virtual memory and get failed application executions even with simple applications such as the SimpleShortestPaths example. While we could force users to set higher vmem-pmem ratios, I think a better option would be to code a configurable margin/fraction of heap usage. In particular, I've always had the habit of setting heap usage at 75% of the total memory available to the container. Doing this, I have had no problems with excessive virtual memory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)