You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/02/18 01:06:11 UTC
[jira] [Resolved] (SPARK-4299) In spark-submit, the driver-memory
value is used for the SPARK_SUBMIT_DRIVER_MEMORY value
[ https://issues.apache.org/jira/browse/SPARK-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-4299.
------------------------------
Resolution: Not a Problem
This may have been fixed along the way, but from examining related issues recently (like https://issues.apache.org/jira/browse/SPARK-5861) I know that yarn-cluster mode does not set the driver process's JVM heap size since it's not the driver.
> In spark-submit, the driver-memory value is used for the SPARK_SUBMIT_DRIVER_MEMORY value
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-4299
> URL: https://issues.apache.org/jira/browse/SPARK-4299
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Virgile Devaux
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> In the spark-submit script, the lines below:
> elif [ "$1" = "--driver-memory" ]; then
> export SPARK_SUBMIT_DRIVER_MEMORY=$2
> are wrong: spark-submit is not the process that will handle the driver when you're in yarn-cluster mode. So, when I lanch spark-submit on a light server with only 2Gb of memory and want to allocate 4gb of memory to the driver (that will run in the ressource manager on a big fat yarn server with, say, 64Gb of RAM) spark submit fails with a OutOfMemory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org