Re: Questions about Aurora scheduling policy

[Riccardo Poggi <ri...@cern.ch>]

Hi Stephan,
thanks for your suggestions.

> have you looked into the quota feature and the updater settings? I have the impression that a combination of both is what you are looking for.
I haven't played with quota yet. If I'm not mistaken updates are for 
services only right?

> Regarding your requirement to have no resource constraints, have you considered starting a certain set of Mesos slaves defaulting to the posix isolators for CPU and RAM? Those posix isolators don't constraint your task at all, i.e., you can always use as much memory as you want or is physically available on the host.
Thanks for the pointer, yes, that's what I'm trying to run.

Cheers,
    Riccardo


On 11/25/2015 01:51 PM, Erb, Stephan wrote:
> Hi Riccardo,
>
> have you looked into the quota feature and the updater settings? I have the impression that a combination of both is what you are looking for.
>
> Regarding your requirement to have no resource constraints, have you considered starting a certain set of Mesos slaves defaulting to the posix isolators for CPU and RAM? Those posix isolators don't constraint your task at all, i.e., you can always use as much memory as you want or is physically available on the host.
>
> Regards,
> Stephan
> ________________________________________
> From: Riccardo Poggi<ri...@cern.ch>
> Sent: Tuesday, November 24, 2015 9:14 PM
> To:dev@aurora.apache.org
> Subject: Re: Questions about Aurora scheduling policy
>
> Thanks Bill,
>
>> I will answer your questions directly, but it may
>> also make sense to have a higher-level discussion about your requirements
>> to possibly offer alternative approaches.
> Sure, that sounds very interesting.
>
> The main scheduling constraint in our system is that it cannot tolerate
> an undefined pending period, it needs at least some fast-feedback.
>
> Example scenario: the system asks to launch N instances of a given
> process with a specified set of resource constraints, now the scheduler
> should either
> - start them all successfully
> - start only a part of them
>       + because of resource saturation for example
> - fail to start them
>       + because those resources are not at all available in the farm, or
> they are effectively busy, or ...
> without queuing, but just returning the operation result.
>
>
> There are also other, more functional, requirements. The most
> interesting ones probably are:
> * Hooks. For good integration with the infrastructure, like publishing
> of information, access management control, and so on...
> * Fault tolerance and Error recovery. A failure in either the scheduler
> or executor shall not take down the managed processes and shall recover
> it after a restart for example.
> * Ownership. Possibility to set uid owner of the underlying processes.
> * Notification system. Provide informations about the jobs/processes
> status (both in pull or push mode)
>
>
>> 3. Would it possible to have Aurora handle tasks with no resource
>>> constraints?
>> [...] but the real question is:
>> what behavior do you want from scheduling without accounting?
> Yes, that is indeed the question. What would be the default scheduling
> behaviour in case of resource abundance?
> Would it be possible to impose a "spreading" constraint on a subset of
> the farm that a-priori I know is able to handle a defined set of jobs?
>
>
> Cheers,
>      Riccardo
>
>
> On 11/24/2015 06:38 AM, Bill Farner wrote:
>> Welcome, happy to help!  I will answer your questions directly, but it may
>> also make sense to have a higher-level discussion about your requirements
>> to possibly offer alternative approaches.
>>
>> 1. Would it possible to subscribe to state change for a given job/task and
>>> receive notifications?
>> Not today, but i'm very open to the idea.  I think there are some cool
>> things you could implement with this behavior.  A concern i often have,
>> though, is that consumers of this data cannot handle cases where an event
>> fails to be delivered (i.e. they want a replica of the scheduler's state).
>> At any rate, i'd love to offer this behavior!
>>
>> 2. Would it possible to set a Pending time-out for tasks that take too long
>>> to be Assigned?
>> Not currently.  You could implement this by polling the API and killing
>> tasks that took too long to schedule.  This would allow you to decide how
>> to react (if at all).
>>
>> 3. Would it possible to have Aurora handle tasks with no resource
>>> constraints?
>> No.  Both Aurora and Mesos require CPU and memory to be specified for
>> tasks.  A client of Aurora could choose defaults, but the real question is:
>> what behavior do you want from scheduling without accounting?
>>
>>
>> On Mon, Nov 23, 2015 at 5:34 PM, Riccardo Poggi<ri...@cern.ch>
>> wrote:
>>
>>> Hello,
>>>
>>> In order to introduce Aurora into our distributed system we would like to
>>> have it slowly, and hopefully transparently, replace what is currently the
>>> process manger component. To do that it would have to programmatically
>>> interface with other parts of the system that are, at the moment, taking
>>> care of what it could be considered the "active" orchestration.
>>>
>>> I've tried Aurora and looked at the docs, but I'm still left with some
>>> open questions:
>>>
>>> 1. Would it possible to subscribe to state change for a given job/task and
>>> receive notifications?
>>>
>>> 2. Would it possible to set a Pending time-out for tasks that take too
>>> long to be Assigned?
>>>
>>> 3. Would it possible to have Aurora handle tasks with no resource
>>> constraints?
>>>
>>>
>>> Thanks,
>>>     Riccardo
>>>