[jira] [Commented] (MESOS-8209) mesos master should revoke offers when executor state changes

["Vinod Kone (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250547#comment-16250547 ] 

Vinod Kone commented on MESOS-8209:
-----------------------------------

```
However, if you start tasks fast enough, the agents can fill up with executors, making it appear as there are no resources available for the scheduler to use. Ive seen this on r4.4xlarge machines on aws with executors that consume 0.1 cpus, 32mb mem where the entire machine will be appear to be filled with executors according to the master resource offers. The executors are exiting (just after the task finishes), but the resources are not reclaimed because the master does not revoke the outstanding resource offers to reflect the change.
```

I don't quite follow this. Master reclaims/recovers executor's resources as soon as the agent notifies of the executor's termination. It should've no bearing on whether a scheduler is holding onto the offers. In other words, what scheduler is holding onto is offered but unallocated resources; what master recovers resources from on tasks/executor termination is allocated resources.

Can you share some logs that show the issue?

> mesos master should revoke offers when executor state changes
> -------------------------------------------------------------
>
>                 Key: MESOS-8209
>                 URL: https://issues.apache.org/jira/browse/MESOS-8209
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Jack Crawford
>            Assignee: Vinod Kone
>
> Currently, the mesos master does not revoke offers when the number of executors on an agent decreases. This is a problem under certain conditions, such when running a workflow that starts lots of small tasks on agents, with a one executor per task model, a master that does not revoke resources after a set amount of time, and a scheduler that does not reject resources.
> The problem is that when running a mono-scheduler framework (which you might want to do to easily enforce authentication requirements, have a full view of all scheduled tasks, etc), in order to respond instantly when new tasks come in I have the scheduler simply hang on to all resource offers it receives, and the master is set to never revoke offers. This way the scheduler always has a pool of resources to quickly service new requests as they come in.
> However, if you start tasks fast enough, the agents can fill up with executors, making it appear as there are no resources available for the scheduler to use. Ive seen this on r4.4xlarge machines on aws with executors that consume 0.1 cpus, 32mb mem where the entire machine will be appear to be filled with executors according to the master resource offers. The executors are exiting (just after the task finishes), but the resources are not reclaimed because the master does not revoke the outstanding resource offers to reflect the change.
> You can replicate this pretty easily if you schedule tasks that finish instantly with a 1-1 executor to task ratio. I find that if I schedule ~1000 tasks this way on a single r4.4xlarge machine, usually 600-700 will finish before all the resource offers to the scheduler fill up and the agent appears to be "full" of executors.
> Changing the scheduler/master to periodically reject/revoke resources fixes the problem.
> My suggestion is for the master to revoke and reissue resource offers when the executor count changes on an agent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)