You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Andrew Ash (JIRA)" <ji...@apache.org> on 2014/05/19 20:40:38 UTC

[jira] [Created] (SPARK-1882) Support dynamic memory sharing in Mesos

Andrew Ash created SPARK-1882:
---------------------------------

             Summary: Support dynamic memory sharing in Mesos
                 Key: SPARK-1882
                 URL: https://issues.apache.org/jira/browse/SPARK-1882
             Project: Spark
          Issue Type: Improvement
          Components: Mesos
    Affects Versions: 1.0.0
            Reporter: Andrew Ash


Fine grained mode Mesos currently supports sharing CPUs very well, but requires that memory be pre-partitioned according to the executor memory parameter.  Mesos supports dynamic memory allocation in addition to dynamic CPU allocation, so we should utilize this feature in Spark.

See below where when the Mesos backend accepts a resource offer it only checks that there's enough memory to cover sc.executorMemory, and doesn't ever take a fraction of the memory available.  The memory offer is accepted all or nothing from a pre-defined parameter.

Coarse mode:
https://github.com/apache/spark/blob/3ce526b168050c572a1feee8e0121e1426f7d9ee/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala#L208

Fine mode:
https://github.com/apache/spark/blob/a5150d199ca97ab2992bc2bb221a3ebf3d3450ba/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L114



--
This message was sent by Atlassian JIRA
(v6.2#6252)