You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tom Hubregtsen (JIRA)" <ji...@apache.org> on 2016/04/04 15:58:25 UTC
[jira] [Created] (SPARK-14367) spark.memory.useLegacyMode=true in
1.6 does not yield the same memory behavior as in 1.3
Tom Hubregtsen created SPARK-14367:
--------------------------------------
Summary: spark.memory.useLegacyMode=true in 1.6 does not yield the same memory behavior as in 1.3
Key: SPARK-14367
URL: https://issues.apache.org/jira/browse/SPARK-14367
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.6.0
Environment: Ubuntu 15.10 with ibm-java-ppc64le-80.
Reporter: Tom Hubregtsen
Priority: Minor
Hi,
I am trying to get the same memory behavior in Spark 1.6 as I had in Spark 1.3 with default settings.
I set
--driver-java-options "--Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6 -Dspark.storage.unrollFraction=0.2"
in Spark 1.6.
But the numbers don't add up. For instance:
--driver-java-options "-Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1"
in Spark 1.3.1 leads to:
16/03/29 14:47:36 INFO MemoryStore: MemoryStore started with capacity 46.1 MB
The same in Spark 1.6.0 with -Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1.
16/03/29 14:50:55 INFO MemoryStore: MemoryStore started with capacity 92.2 MB
If I then increase both fractions to 0.2, the numbers of the MemoryStore both double (as one would expect), but that means there is still a 2x difference in allocated memory between Spark 1.3 and Spark 1.6. So my question:
I believe a parameter that reads
spark.memory.useLegacyMode=true
should yield the *exact* memory behavior as in the Legacy version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org