[jira] [Commented] (SPARK-14367) spark.memory.useLegacyMode=true in 1.6 does not yield the same memory behavior as Spark 1.3

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225909#comment-15225909 ] 

Sean Owen commented on SPARK-14367:
-----------------------------------

The current function that would determine this value is ...

{code}
  /**
   * Return the total amount of memory available for the storage region, in bytes.
   */
  private def getMaxStorageMemory(conf: SparkConf): Long = {
    val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
    val memoryFraction = conf.getDouble("spark.storage.memoryFraction", 0.6)
    val safetyFraction = conf.getDouble("spark.storage.safetyFraction", 0.9)
    (systemMaxMemory * memoryFraction * safetyFraction).toLong
  }
{code}

It does return the correct value (46.1MB) if spark.storage.memoryFraction=0.1 (and your runtime memory is 512m).

Are you certain that spark.storage.memoryFraction=0.1 in the 1.6 case, and that the executor memory hasn't changed to 1024m? It looks like it does just what you'd expect it to, code-wise.

> spark.memory.useLegacyMode=true in 1.6 does not yield the same memory behavior as Spark 1.3
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-14367
>                 URL: https://issues.apache.org/jira/browse/SPARK-14367
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.6.0
>         Environment: Ubuntu 15.10 with ibm-java-ppc64le-80. 
>            Reporter: Tom Hubregtsen
>            Priority: Minor
>              Labels: backwards-compatibility
>
>  Hi,
> I am trying to get the same memory behavior in Spark 1.6 as I had in Spark 1.3 with default settings.
> I set
> --driver-java-options "--Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6 -Dspark.storage.unrollFraction=0.2"
> in Spark 1.6.
> But the numbers don't add up. For instance:
> --driver-java-options "-Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1"
> in Spark 1.3.1 leads to:
> 16/03/29 14:47:36 INFO MemoryStore: MemoryStore started with capacity 46.1 MB
> The same in Spark 1.6.0 with -Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1.
> 16/03/29 14:50:55 INFO MemoryStore: MemoryStore started with capacity 92.2 MB
> If I then increase both fractions to 0.2, the numbers of the MemoryStore both double (as one would expect), but that means there is still a 2x difference in allocated memory between Spark 1.3 and Spark 1.6. So my question:
> I believe a parameter that reads
> spark.memory.useLegacyMode=true
> should yield the *exact* memory behavior as in the Legacy version. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org