[jira] [Commented] (HADOOP-11364) [Java 8] Over usage of virtual memory

["Mohammad Kamrul Islam (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238716#comment-14238716 ] 

Mohammad Kamrul Islam commented on HADOOP-11364:
------------------------------------------------

My findings and quick resolutions:
By default, Java 8 allocates extra virtual memory then Java 7. However, we can control the non-heap memory usage by limiting the maximum allowed values for some JVM  parameters  such as  "-XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256"

For M/R based job (such as Pig, Hive etc), user can pass the following  JVM -XX parameters as part of mapreduce.reduce.java.opts or mapreduce.map.java.opts
{noformat}
mapreduce.reduce.java.opts  '-XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m -Xmx1536m -Xms512m -Djava.net.preferIPv4Stack=true'
{noformat}

Similarly for Spark job, we need to pass the same parameters in the Spark AM/master and executor. Spark community is working on the ways to pass these type of parameters easily. In Spark-1.1.0, user can pass it for spark-cluster based job submission as follows. For general job submission, user has to wait until https://issues.apache.org/jira/browse/SPARK-4461 is released.
{noformat}
spark.driver.extraJavaOptions = -XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m
{noformat}

For Spark executor, pass the following.
{noformat} 
spark.executor.extraJavaOptions = -XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m
{noformat}

 These parameters can be set in conf/spark-defaults.conf as well.

> [Java 8] Over usage of virtual memory
> -------------------------------------
>
>                 Key: HADOOP-11364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11364
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>            Assignee: Mohammad Kamrul Islam
>
> In our Hadoop 2 + Java8 effort , we found few jobs are being Killed by Hadoop due to excessive virtual memory allocation.  Although the physical memory usage is low.
> The most common error message is "Container [pid=??,containerID=container_??] is running beyond virtual memory limits. Current usage: 365.1 MB of 1 GB physical memory used; 3.2 GB of 2.1 GB virtual memory used. Killing container."
> We see this problem for MR job as well as in spark driver/executor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)