[jira] [Created] (MESOS-4616) Support specifying a preferred host with a Resource Request

["Jagadish (JIRA)" <ji...@apache.org>]

Jagadish created MESOS-4616:
-------------------------------

             Summary: Support specifying a preferred host with a Resource Request
                 Key: MESOS-4616
                 URL: https://issues.apache.org/jira/browse/MESOS-4616
             Project: Mesos
          Issue Type: Story
            Reporter: Jagadish


When stateful services like Apache Samza, Kafka must be restarted with using Mesos, the framework must have a way of specifying a preferred host with the request. 

More background:
I work on Apache Samza , a distributed stream processing framework. Currently Samza supports only Yarn as a resource manager. (there have been requests to run Samza with mesos). A cluster (200 nodes 'ish) runs many Samza Jobs (about 3500). Each Samza Job has its own framework that requests resources (containers) for the job to run. Each such container uses GBs of local state  . When such a container(resource) is started on a different host by the framework, the local state must be re-bootstrapped.  (this results in a long bootstrap time, which is essentially down time).

The same is true for Apache Kafka, a distributed pub-sub logging system.  When a Kafka broker must be restarted by the framework, it should ideally be re-started on the same host. (otherwise, each broker has to re-bootstrap several GBs of logs from its peers before it can start to service a request.)

I'm sure many stateful services have similar requirements.

My framework can update/ filter resource offers, but with many frameworks (thousands) on several nodes, I'm concerned about the wait-time for a particular available resource to be offered to a framework that needs it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)