SparkConf does not work for spark.driver.memory

[wgtmac <us...@gmail.com>]

Hi

I'm using spark 1.5.1. But I encountered a problem using SparkConf to set
spark.driver.memory in yarn-cluster mode.

Example 1:

In the code, I did following:
val sc = new SparkContext(new
SparkConf().setAppName("test").set("spark.driver.memory", "4g"))

And used following command to submit job:

YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit \
--class path.to.className \
--jars jarName \
--queue default \
--num-executors 4 \
--conf spark.eventLog.overwrite=true \
--conf spark.eventLog.enabled=true \
--conf spark.eventLog.dir=hdfs://localhost:port \
--conf spark.yarn.historyServer.address=http://host:port 

Although in the webUI spark.driver.memory is 4g but actually driver memory
is 1g (default value)
And my job failed due to shortage of driver memory which throws exceeding GC
limit exception.

Example 2:
Almost same configuration except I added "--driver-memory 2g" at the end of
the command.
Then in the webUI spark.driver.memory is 4g but actually driver memory is 2g
now.
My job succeeded which proves the driver memory is different with example 1.

So my question is that: In yarn-cluster mode, is it ineffective to use
SparkConf to set spark.driver.memory?
Thanks!





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkConf-does-not-work-for-spark-driver-memory-tp26270.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: SparkConf does not work for spark.driver.memory

[Marcelo Vanzin <va...@cloudera.com>]

On Thu, Feb 18, 2016 at 10:26 AM, wgtmac <us...@gmail.com> wrote:
> In the code, I did following:
> val sc = new SparkContext(new
> SparkConf().setAppName("test").set("spark.driver.memory", "4g"))

You can't set the driver memory like this, in any deploy mode. When
that code runs, the driver is already running, so there's no way to
modify the JVM's command line options at that time.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org