You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Sean Owen (Jira)" <ji...@apache.org> on 2019/09/09 16:26:00 UTC

[jira] [Resolved] (SPARK-25261) Standardize the default units of spark.driver|executor.memory

     [ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-25261.
-------------------------------
       Fix Version/s: 2.4.0
    Target Version/s:   (was: 3.0.0)
            Assignee: huangtengfei
          Resolution: Fixed

Resolved by https://github.com/apache/spark/pull/22252

> Standardize the default units of spark.driver|executor.memory
> -------------------------------------------------------------
>
>                 Key: SPARK-25261
>                 URL: https://issues.apache.org/jira/browse/SPARK-25261
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes, Spark Core, YARN
>    Affects Versions: 2.3.0
>            Reporter: huangtengfei
>            Assignee: huangtengfei
>            Priority: Minor
>             Fix For: 2.4.0
>
>
> From  [SparkContext|https://github.com/ivoson/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L464] and [SparkSubmitCommandBuilder|https://github.com/ivoson/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L265]，we can see that spark.driver.memory and spark.executor.memory are parsed as bytes if no units specified. But in the doc, they are described as mb in default, which may lead to some misunderstanding.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org