[jira] [Created] (AMBARI-13719) MAPREDUCE2 service check fails sporadically with JDK1.8

["Alexander Denissov (JIRA)" <ji...@apache.org>]

Alexander Denissov created AMBARI-13719:
-------------------------------------------

             Summary: MAPREDUCE2 service check fails sporadically with JDK1.8
                 Key: AMBARI-13719
                 URL: https://issues.apache.org/jira/browse/AMBARI-13719
             Project: Ambari
          Issue Type: Bug
            Reporter: Alexander Denissov


Used PHD Ambari 2.1.2 with JDK 1.8 to create a cluster using blueprints. Cluster in AWS using r3.large instances (~17G of RAM).

Seems that stack advisor is not used during a blueprint deployment, so the services take default configurations. For MAPREDUCE2 the value of mapreduce.map.memory.mb is set to "512" and mapreduce.map.java.opts is set to "-Xmx410m"

Service checks are not run by default during blueprint deployment. When a service check REST API is called, the service check fails with mapper task running out of memory:
2015-11-03 18:53:56,256 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
       at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:986)
       at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)

When connecting to the cluster via UI and re-running the service check, it sometimes succeeds and sometimes fails again.

This behavior is observed when using JDK1.8 -- the same environment, blueprint and tests work well with JDK 1.7

The behavior is observed with JDK 1.8 with the blueprint containing many services (HBase, Hive, etc). A blueprint with just HDFS/YARN/MR2/ZK seems not to have this issue on JDK 1.8. 

Manually increasing the memory parameters (or adding configurations to the blueprint) seem to resolve the problem:
"mapred-site":{  
            "mapreduce.map.java.opts":"-Xmx819m",
            "mapreduce.map.memory.mb":"1024"
 }

However, we should get to the root cause of why using JDK1.8 causes OOM errors on default configuration. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)