Executor Resource Requirements

[Christopher Ketchum <ck...@ucsc.edu>]

Hi,

I am working on a framework for mesos, and I have a question about resource offers and executor resource requirements. We are using a long-lived executor, and one executor per slave. When the master re-offers a slave’s resources to the framework, and that slave already has an executor running on it, we see that the offer is for the slave’s resources minus the executor requirements.

However, when a slave does not have a running executor, obviously mesos does not take the executor’s requirements into account for that offer. This leads to some problems though, because if, for example, our slave has 1 cpu we would get an offer for 1 cpu. If we then launch a task with 1 cpu on that offer, mesos won’t run that task and sets it’s state to ‘TASK_ERROR’, with message 'Task uses more resources cpus(*)1.1 …. than available cpus(*)1’. Our executor uses 0.1 cpu, so it seems like mesos aggregates the executor and the task requirements, but only if there is no executor currently running. 

My question is, is there an easy way to know if there is an executor currently running on a slave? It seems like we want to treat offers differently depending on wether the executor is already taken into account, but how can we tell? Is it up to our framework to keep track of this?

Thanks,
Christopher

Re: Executor Resource Requirements

[Benjamin Mahler <be...@gmail.com>]

The offer will contain the running executors on that slave:
https://github.com/apache/mesos/blob/0.22.1/include/mesos/mesos.proto#L619

However, there are a few bugs to note if you're planning to use this in any
kind of production setup:

(1) If an executor exits, or a new one is started, we won't rescind the
offer to give you an updated list of executor ids.
(2) In some cases, he master may not accurately know whether executors are
running: https://issues.apache.org/jira/browse/MESOS-1961

On Wed, Jun 17, 2015 at 5:37 PM, Vinod Kone <vi...@gmail.com> wrote:

>
> On Wed, Jun 17, 2015 at 12:19 PM, Christopher Ketchum <ck...@ucsc.edu>
> wrote:
>
>> My question is, is there an easy way to know if there is an executor
>> currently running on a slave? It seems like we want to treat offers
>> differently depending on wether the executor is already taken into account,
>> but how can we tell? Is it up to our framework to keep track of this?
>>
>
> That is correct. The framework is expected to keep track of this.
>
>

Re: Executor Resource Requirements

[Vinod Kone <vi...@gmail.com>]

On Wed, Jun 17, 2015 at 12:19 PM, Christopher Ketchum <ck...@ucsc.edu>
wrote:

> My question is, is there an easy way to know if there is an executor
> currently running on a slave? It seems like we want to treat offers
> differently depending on wether the executor is already taken into account,
> but how can we tell? Is it up to our framework to keep track of this?
>

That is correct. The framework is expected to keep track of this.