[jira] [Commented] (AURORA-1726) Create support for using multiple executors in the Scheduler

["Renan DelValle (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383446#comment-15383446 ] 

Renan DelValle commented on AURORA-1726:
----------------------------------------

Currently have a working implementation of this that does its best not to get in the way of how things are run with thermos.

One concern I have is about preemption. As far as I can see, the resources available for preemption are calculated using the resources being used by the victim task + executor overhead.

This could result in a corner case that may or may not manifest itself. It is not exclusive to using multiple executors but may be magnified by the feature if the resource overhead is changed and the scheduler is restarted with a larger resource overhead.

Maybe the more experienced devs can help me understand if this scenario is possible:
Overhead for thermos is set to C cpus and R ram
task A is submitted with A[cpus] cpus, A[ram] ram, and A[disk] disk. 
task A begins to run with A[cpus] + C cpus, A[ram] + R ram, and A[disk] disk.

Overhead is changed to C' cpus and R' ram. Scheduler is restarted and running tasks are reconciled.

task B is submitted to the scheduler with B[cpus, B[ram] and B[disk].

Preemption calculations begin. Since the calculations take into account the current overhead set,  the resources available for pre-emption are incorrectly calculated to be:
A[cpus] + C', A[ram] + R', A[disk]
When they should be using the values used at the time of scheduling:
A[cpus] + C, A[ram] + R, A[disk]

If this scenario is possible, we should come up with a suitable solution to this issue which may involve storing the overhead used for tasks at the time of running them.

> Create support for using multiple executors in the Scheduler
> ------------------------------------------------------------
>
>                 Key: AURORA-1726
>                 URL: https://issues.apache.org/jira/browse/AURORA-1726
>             Project: Aurora
>          Issue Type: Sub-task
>          Components: Scheduler
>            Reporter: Renan DelValle
>            Assignee: Renan DelValle
>
> Allow a single Aurora scheduler to schedule tasks on Mesos with different executors. Configuration for executors will be server side and loaded at the time the Scheduler is started. Users may specify the executor they wish to use by specifying the name executor they wish their task to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)