[jira] [Created] (SPARK-14822) Add lazy executor startup to Mesos Scheduler

["Michael Gummelt (JIRA)" <ji...@apache.org>]

Michael Gummelt created SPARK-14822:
---------------------------------------

             Summary: Add lazy executor startup to Mesos Scheduler
                 Key: SPARK-14822
                 URL: https://issues.apache.org/jira/browse/SPARK-14822
             Project: Spark
          Issue Type: Task
          Components: Mesos
            Reporter: Michael Gummelt


As we deprecate fine-grained mode, we need to make sure we have alternative solutions for its benefits.

Its two benefits are:

0. lazy executor startup
  In fine-grained mode, executors are brought up only as tasks are scheduled.  This means that a user doesn't have to set {{spark.cores.max}} to ensure that the app doesn't consume all resources in the cluster.

1. relinquishing cores
  As Spark tasks terminate, the mesos task it was bound to terminates as well, thus relinquishing the cores assigned to it.

I'd like to add {{0.}} to coarse-grained mode, possibly enabled with a configuration param.  If https://issues.apache.org/jira/browse/MESOS-1279 ever happens, we can add {{1.}} as well.

cc [~tnachen] [~dragos] [~skonto] [~andrewor14]




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org