You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Ash (JIRA)" <ji...@apache.org> on 2014/11/10 18:48:33 UTC
[jira] [Commented] (SPARK-1882) Support dynamic memory sharing in
Mesos
[ https://issues.apache.org/jira/browse/SPARK-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205068#comment-14205068 ]
Andrew Ash commented on SPARK-1882:
-----------------------------------
Another major issue is to dynamically scale hardware usage as jobs come online, you need to scale both memory and cores up and down.
Scaling cores is pretty easy via starting/stopping JVM threads (as Spark does already) but scaling memory in a JVM is more difficult. Scaling up is somewhat feasible (set Xms to initial size and Xmx to something huge, then JVM scaling up happens automatically and could be moderated by the block manager self-regulating), but scaling memory down is hard.
I started a thread on jdk9-dev here and think this will be more possible with JVMs in the future but it's not quite there yet: http://mail.openjdk.java.net/pipermail/jdk9-dev/2014-August/001237.html
Until scaling a JVM heap size down is possible this may just be the best that's possible -- scaling cores at will and memory up but not down.
> Support dynamic memory sharing in Mesos
> ---------------------------------------
>
> Key: SPARK-1882
> URL: https://issues.apache.org/jira/browse/SPARK-1882
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
> Affects Versions: 1.0.0
> Reporter: Andrew Ash
>
> Fine grained mode Mesos currently supports sharing CPUs very well, but requires that memory be pre-partitioned according to the executor memory parameter. Mesos supports dynamic memory allocation in addition to dynamic CPU allocation, so we should utilize this feature in Spark.
> See below where when the Mesos backend accepts a resource offer it only checks that there's enough memory to cover sc.executorMemory, and doesn't ever take a fraction of the memory available. The memory offer is accepted all or nothing from a pre-defined parameter.
> Coarse mode:
> https://github.com/apache/spark/blob/3ce526b168050c572a1feee8e0121e1426f7d9ee/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala#L208
> Fine mode:
> https://github.com/apache/spark/blob/a5150d199ca97ab2992bc2bb221a3ebf3d3450ba/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L114
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org