[jira] [Commented] (SPARK-32293) Inconsistent default unit between Spark memory configs and JVM option

["Takeshi Yamamuro (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-32293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169946#comment-17169946 ] 

Takeshi Yamamuro commented on SPARK-32293:
------------------------------------------

Hi, [~attilapiros]. You don't need to set the target version, so I removed it.

> Inconsistent default unit between Spark memory configs and JVM option
> ---------------------------------------------------------------------
>
>                 Key: SPARK-32293
>                 URL: https://issues.apache.org/jira/browse/SPARK-32293
>             Project: Spark
>          Issue Type: Bug
>          Components: Documentation, Spark Core
>    Affects Versions: 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 3.0.0, 3.0.1, 3.1.0
>            Reporter: Attila Zsolt Piros
>            Priority: Major
>
> Spark's maximum memory can be configured in several ways:
> - via Spark config
> - command line argument
> - environment variables 
> Both for executors and for the driver the memory can be configured separately. All of these are following the format of JVM memory configurations in a way they are using the very same size unit suffixes ("k", "m", "g" or "t") but there is an inconsistency regarding the default unit. When no suffix is given then the given amount is passed as it is to the JVM (to the -Xmx and -Xms options) where this memory options are using bytes as a default unit, for this please see the example [here|https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html]:
> {noformat}
> The following examples show how to set the maximum allowed size of allocated memory to 80 MB using various units:
> -Xmx83886080 
> -Xmx81920k 
> -Xmx80m
> {noformat}
> Although the Spark memory config default suffix unit is "m".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org