[jira] [Created] (REEF-102) Explore new ways of using Mesos Executor and Mesos Task

["John Yang (JIRA)" <ji...@apache.org>]

John Yang created REEF-102:
------------------------------

             Summary: Explore new ways of using Mesos Executor and Mesos Task
                 Key: REEF-102
                 URL: https://issues.apache.org/jira/browse/REEF-102
             Project: REEF
          Issue Type: Improvement
          Components: REEF-Runtime-Mesos
            Reporter: John Yang
            Priority: Minor


To compare REEF's Mesos Executor&Task usage with Spark's:

Spark on Mesos Default Mode

* Mesos Executor: Spark Executor (implements Mesos Executor)
* Mesos Task: Spark Task (launched via Mesos launchTasks() interface)

Spark on Mesos Coarse-Grained Mode

* Mesos Executor: Default Mesos Executor
* Mesos Task: Spark Executor (does not implement any Mesos-provided class)

Current REEF on Mesos
* Mesos Executor: REEF Executor (implements Mesos Executor)
* Mesos Task: Nothing (we assume it holds onto the resources until REEF Evaluator release)

As you can see, our approach is not as clear-cut as Spark's approaches. As a result we have ad-hoc/redundant pieces of code(marked as TODOs) in REEFScheduler and REEFExecutor. 

Spark Default Mode-like approach is too much work for us as we'd have to create another version(or a module) of REEF Evaluator that implements Mesos Executor. Crucially, we'd like to have REEF Evaluators to hold onto the initially requested resources until they are released.

An approach similar to Spark Coarse-Grained Mode takes might be worth taking a stab at. We can run a process that does not implement Mesos Executor as a Mesos Task upon REEF Evaluator request. It can then receive Evaluator launch/release command from REEF Driver. I think we can have cleaner code if this approach is taken.

Note that the difference from Spark's path would be that the process launched as a Mesos Task acts as a bridge for launching/releasing REEF Evaluator whereas in Spark, Spark Executor is simply launched as a Mesos Task. This is because REEF has a user-configured two-step events(Resource Request & Resource Launch) where it is not possible to generate EvaluatorLaunchCommand upon Resource Request, leading to the use of a separate process in obtaining Mesos resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)