You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by andrewor14 <gi...@git.apache.org> on 2015/12/02 00:49:00 UTC
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/10081
[SPARK-12081] Make unified memory manager work with small heaps
The existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM.
**New proposal.** We now reserve 300MB before taking the 75% for execution and storage. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark unified-memory-small-heaps
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10081.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10081
----
commit 15718776665db6ebff463587e726bb1e704dabc7
Author: Andrew Or <an...@databricks.com>
Date: 2015-12-01T23:44:34Z
Carve out 300MB reserved memory
The new space used by storage and execution will be calculated
by (JVM size - 300MB) * 75%, the `spark.memory.fraction`.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161165555
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161143882
**[Test build #2141 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2141/consoleFull)** for PR 10081 at commit [`1571877`](https://github.com/apache/spark/commit/15718776665db6ebff463587e726bb1e704dabc7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161165248
**[Test build #47025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47025/consoleFull)** for PR 10081 at commit [`2e9bc4d`](https://github.com/apache/spark/commit/2e9bc4de1e53b722406750b22964b0f1d96fbef8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161172467
Thanks I'm merging into master 1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161175141
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47014/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161134407
@davies @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161143825
**[Test build #2139 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2139/consoleFull)** for PR 10081 at commit [`1571877`](https://github.com/apache/spark/commit/15718776665db6ebff463587e726bb1e704dabc7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161165558
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47025/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161146291
**[Test build #47025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47025/consoleFull)** for PR 10081 at commit [`2e9bc4d`](https://github.com/apache/spark/commit/2e9bc4de1e53b722406750b22964b0f1d96fbef8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161163513
**[Test build #2139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2139/consoleFull)** for PR 10081 at commit [`1571877`](https://github.com/apache/spark/commit/15718776665db6ebff463587e726bb1e704dabc7).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161138578
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161143969
**[Test build #2140 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2140/consoleFull)** for PR 10081 at commit [`1571877`](https://github.com/apache/spark/commit/15718776665db6ebff463587e726bb1e704dabc7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161175097
**[Test build #47014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47014/consoleFull)** for PR 10081 at commit [`1571877`](https://github.com/apache/spark/commit/15718776665db6ebff463587e726bb1e704dabc7).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10081#discussion_r46363403
--- Diff: core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala ---
@@ -130,6 +130,8 @@ private[spark] class UnifiedMemoryManager private[memory] (
object UnifiedMemoryManager {
+ private val RESERVED_SYSTEM_MEMORY_BYTES = 300 * 1024 * 1024
--- End diff --
you should document what this is and update the classdoc for UnifiedMemoryManager
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10081#discussion_r46432025
--- Diff: core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala ---
@@ -130,6 +130,12 @@ private[spark] class UnifiedMemoryManager private[memory] (
object UnifiedMemoryManager {
+ // Set aside a fixed amount of memory for non-storage, non-execution purposes.
+ // This serves a function similar to `spark.memory.fraction`, but guarantees that we reserve
+ // sufficient memory for the system even for small heaps. E.g. if we have a 1GB JVM, then
+ // the memory used for execution and storage will be (1024 - 300) * 0.75 = 543MB by default.
+ private val RESERVED_SYSTEM_MEMORY_BYTES = 300 * 1024 * 1024
--- End diff --
For non-small heap, the above memory doesn't need to be reserved, right ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10081
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by jcsalterego <gi...@git.apache.org>.
Github user jcsalterego commented on a diff in the pull request:
https://github.com/apache/spark/pull/10081#discussion_r53967310
--- Diff: core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala ---
@@ -144,8 +150,16 @@ object UnifiedMemoryManager {
* Return the total amount of memory shared between execution and storage, in bytes.
*/
private def getMaxMemory(conf: SparkConf): Long = {
- val systemMaxMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
+ val systemMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
+ val reservedMemory = conf.getLong("spark.testing.reservedMemory",
+ if (conf.contains("spark.testing")) 0 else RESERVED_SYSTEM_MEMORY_BYTES)
+ val minSystemMemory = reservedMemory * 1.5
+ if (systemMemory < minSystemMemory) {
+ throw new IllegalArgumentException(s"System memory $systemMemory must " +
+ s"be at least $minSystemMemory. Please use a larger heap size.")
--- End diff --
This comes out as scientific notation, which made me laugh.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161135468
**[Test build #47014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47014/consoleFull)** for PR 10081 at commit [`1571877`](https://github.com/apache/spark/commit/15718776665db6ebff463587e726bb1e704dabc7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12081] Make unified memory manager work...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10081#issuecomment-161175139
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org