You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "John Vines (JIRA)" <ji...@apache.org> on 2016/01/05 17:03:39 UTC
[jira] [Created] (SPARK-12650) No means to specify Xmx settings for
SparkSubmit in yarn-cluster mode
John Vines created SPARK-12650:
----------------------------------
Summary: No means to specify Xmx settings for SparkSubmit in yarn-cluster mode
Key: SPARK-12650
URL: https://issues.apache.org/jira/browse/SPARK-12650
Project: Spark
Issue Type: Bug
Affects Versions: 1.5.2
Environment: Hadoop 2.6.0
Reporter: John Vines
Background-
I have an app master designed to do some work and then launch a spark job.
Issue-
If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all, leading to the jvm taking a default heap which is relatively large. This causes a large amount of vmem to be taken, so that it is killed by yarn. This can be worked around by disabling Yarn's vmem check, but that is a hack.
If I run it in yarn-client mode, it's fine as long as my container has enough space for the driver, which is manageable. But I feel that the utter lack of Xmx settings for what I believe is a very small jvm is a problem.
I believe this was introduced with the fix for SPARK-3884
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org