You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/01/14 15:12:20 UTC
[GitHub] srowen commented on a change in pull request #23447:
[SPARK-26462][CORE] Use ConfigEntry for hardcoded configs for execution
categories
srowen commented on a change in pull request #23447: [SPARK-26462][CORE] Use ConfigEntry for hardcoded configs for execution categories
URL: https://github.com/apache/spark/pull/23447#discussion_r247528412
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -187,6 +188,105 @@ package object config {
.checkValue(_ >= 0, "The off-heap memory size must not be negative")
.createWithDefault(0)
+ private[spark] val MEMORY_STORAGE_FRACTION = ConfigBuilder("spark.memory.storageFraction")
+ .doc("Amount of storage memory immune to eviction, expressed as a fraction of the " +
+ "size of the region set aside by spark.memory.fraction. The higher this is, the " +
+ "less working memory may be available to execution and tasks may spill to disk more " +
+ "often. Leaving this at the default value is recommended. ")
+ .doubleConf
+ .createWithDefault(0.5)
+
+ private[spark] val MEMORY_FRACTION = ConfigBuilder("spark.memory.fraction")
+ .doc("Fraction of (heap space - 300MB) used for execution and storage. The " +
+ "lower this is, the more frequently spills and cached data eviction occur. " +
+ "The purpose of this config is to set aside memory for internal metadata, " +
+ "user data structures, and imprecise size estimation in the case of sparse, " +
+ "unusually large records. Leaving this at the default value is recommended. ")
+ .doubleConf
+ .createWithDefault(0.6)
+
+ private[spark] val MEMORY_USE_LEGACY_MODE = ConfigBuilder("spark.memory.useLegacyMode")
Review comment:
This needs to be removed now; the legacy mode was deleted recently. Same with `spark.shuffle.memoryFraction`, `spark.storage.memoryFraction`, `spark.storage.unrollFraction`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org