You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/08/30 13:51:00 UTC
[jira] [Updated] (SPARK-25261) Standardize the default units of
spark.driver|executor.memory
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-25261:
------------------------------
Target Version/s: 3.0.0
Component/s: (was: Documentation)
YARN
Spark Core
Kubernetes
Summary: Standardize the default units of spark.driver|executor.memory (was: Update configuration.md, correct the default units of spark.driver|executor.memory)
Really, these properties are parsed differently in different parts of the code, unfortunately. YARN and K8S parse a string without units as MiB, but not spark-submit.
We can fix the docs in the first PR, and then in Spark 3, correct this behavior inconsistency. I would even vote for no longer supporting unit-less strings in any such property, then.
> Standardize the default units of spark.driver|executor.memory
> -------------------------------------------------------------
>
> Key: SPARK-25261
> URL: https://issues.apache.org/jira/browse/SPARK-25261
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core, YARN
> Affects Versions: 2.3.0
> Reporter: huangtengfei
> Priority: Minor
>
> From [SparkContext|https://github.com/ivoson/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L464] and [SparkSubmitCommandBuilder|https://github.com/ivoson/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L265]，we can see that spark.driver.memory and spark.executor.memory are parsed as bytes if no units specified. But in the doc, they are described as mb in default, which may lead to some misunderstanding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org