You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@aurora.apache.org by Jacob Scott <ja...@gmail.com> on 2016/09/27 05:34:05 UTC
roles or attributes for virtual clusters
I have a mesos cluster running on EC2 that I want to split into two
"virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for
blue jobs and red nodes/instances for red jobs.
After reading the documentation I can see two ways of doing this:
1. Set color:red and color:blue attributes on mesos agents[1] and
similarly in Aurora job config [2]
2. Using mesos roles [3], Run one blue Aurora scheduler [4] and one red
scheduler
Is one approach here better, or more in line with best practices? Two
schedulers seems like more operational overhead, but using attributes might
result in a higher offer reject rate (Aurora receiving and rejecting red
offers when trying to run a blue job)?
I appreciate any advice or pointers to relevant resources.
Thanks,
Jacob
[1] http://mesos.apache.org/documentation/latest/attributes-resources/
[2] http://aurora.apache.org/documentation/latest/features/constraints/
[3] http://mesos.apache.org/documentation/latest/roles/
[4]
http://aurora.apache.org/documentation/latest/reference/scheduler-configuration/
Re: roles or attributes for virtual clusters
Posted by Jacob Scott <ja...@gmail.com>.
I *think* attribute-based virtual clusters are the pragmatic approach for
now, but multiple clusters is certainly possible and it makes sense to keep
tabs on the additional complexity agent classification adds.
Thanks for the feedback!
Jacob
On Wed, Sep 28, 2016 at 2:25 PM, Rick Mangi <ri...@chartbeat.com> wrote:
> I wonder if you might be better off just having 2 clusters, it might make
> things easier to manage in the long term.
>
>
> On Sep 28, 2016, at 5:01 PM, Jacob Scott <ja...@gmail.com> wrote:
>
> Thanks much (and thanks for fielding a version of this question in IRC!).
>
> Jacob
>
> On Wed, Sep 28, 2016 at 1:24 PM Zameer Manji <zm...@apache.org> wrote:
>
>> +1
>>
>> I've found the complexity of managing mesos roles to be high. For a fix
>> that has less overhead, I think scheduling on attributes via Aurora's
>> constraint system would be easiest.
>>
>> On Wed, Sep 28, 2016 at 1:20 PM, Jacob Scott <ja...@gmail.com>
>> wrote:
>>
>>> I'm running Chronos and Aurora, with an existing workload which has high
>>> reliability requirements. I'm adding in a new, more experimental workload,
>>> and would like the different workloads to run on different hardware to
>>> reduce risk. Some of the risk comes from me playing fast and loose with
>>> per-job cpu/mem/disk, but I expect it to be easier to run for now with
>>> different agent classes than untangle that ball of wax. There's also some
>>> secondary benefits like better attribution for billing.
>>>
>>> Based on further reading on roles, and the complexity of reserving
>>> resources for roles, I'm currently planning to use attributes -- one
>>> Aurora, one Chronos, one Mesos Master, agents classified via attribute).
>>>
>>>
>>> Thanks,
>>>
>>> Jacob
>>>
>>> On Wed, Sep 28, 2016 at 1:11 PM, Erb, Stephan <
>>> Stephan.Erb@blue-yonder.com> wrote:
>>>
>>>> Hi Jacob,
>>>>
>>>>
>>>>
>>>> could you elaborate why you want to establish two different classes of
>>>> agents? Do those have different hardware specs or is this due to a
>>>> different network setup or something like that?
>>>>
>>>>
>>>>
>>>> Furthermore, do you plan to run other frameworks in addition to Aurora?
>>>>
>>>>
>>>>
>>>> If you only plan to use Aurora for now, I would start with a single
>>>> Aurora scheduler and see how this works out for you. Aurora does not
>>>> immediately reject offers but it holds onto them for a default of 5 minutes
>>>> (IIRC). The single scheduler then has a complete view of all available
>>>> resources and should be able to schedule your jobs rather quickly.
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Stephan
>>>>
>>>>
>>>>
>>>> *From: *Jacob Scott <ja...@gmail.com>
>>>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>>> *Date: *Tuesday 27 September 2016 at 07:34
>>>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>>> *Subject: *roles or attributes for virtual clusters
>>>>
>>>>
>>>>
>>>> I have a mesos cluster running on EC2 that I want to split into two
>>>> "virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for
>>>> blue jobs and red nodes/instances for red jobs.
>>>>
>>>>
>>>>
>>>> After reading the documentation I can see two ways of doing this:
>>>>
>>>> 1. Set color:red and color:blue attributes on mesos agents[1] and
>>>> similarly in Aurora job config [2]
>>>> 2. Using mesos roles [3], Run one blue Aurora scheduler [4] and one
>>>> red scheduler
>>>>
>>>> Is one approach here better, or more in line with best practices? Two
>>>> schedulers seems like more operational overhead, but using attributes might
>>>> result in a higher offer reject rate (Aurora receiving and rejecting red
>>>> offers when trying to run a blue job)?
>>>>
>>>>
>>>>
>>>> I appreciate any advice or pointers to relevant resources.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> Jacob
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [1] http://mesos.apache.org/documentation/latest/attributes-resources/
>>>>
>>>> [2] http://aurora.apache.org/documentation/latest/features/constraints/
>>>>
>>>> [3] http://mesos.apache.org/documentation/latest/roles/
>>>>
>>>> [4] http://aurora.apache.org/documentation/latest/reference/scheduler-
>>>> configuration/
>>>>
>>>>
>>>
>>
>
Re: roles or attributes for virtual clusters
Posted by Rick Mangi <ri...@chartbeat.com>.
I wonder if you might be better off just having 2 clusters, it might make things easier to manage in the long term.
> On Sep 28, 2016, at 5:01 PM, Jacob Scott <ja...@gmail.com> wrote:
>
> Thanks much (and thanks for fielding a version of this question in IRC!).
>
> Jacob
>
> On Wed, Sep 28, 2016 at 1:24 PM Zameer Manji <zmanji@apache.org <ma...@apache.org>> wrote:
> +1
>
> I've found the complexity of managing mesos roles to be high. For a fix that has less overhead, I think scheduling on attributes via Aurora's constraint system would be easiest.
>
> On Wed, Sep 28, 2016 at 1:20 PM, Jacob Scott <jacob.scott@gmail.com <ma...@gmail.com>> wrote:
> I'm running Chronos and Aurora, with an existing workload which has high reliability requirements. I'm adding in a new, more experimental workload, and would like the different workloads to run on different hardware to reduce risk. Some of the risk comes from me playing fast and loose with per-job cpu/mem/disk, but I expect it to be easier to run for now with different agent classes than untangle that ball of wax. There's also some secondary benefits like better attribution for billing.
>
> Based on further reading on roles, and the complexity of reserving resources for roles, I'm currently planning to use attributes -- one Aurora, one Chronos, one Mesos Master, agents classified via attribute).
>
>
> Thanks,
>
> Jacob
>
> On Wed, Sep 28, 2016 at 1:11 PM, Erb, Stephan <Stephan.Erb@blue-yonder.com <ma...@blue-yonder.com>> wrote:
> Hi Jacob,
>
>
>
> could you elaborate why you want to establish two different classes of agents? Do those have different hardware specs or is this due to a different network setup or something like that?
>
>
>
> Furthermore, do you plan to run other frameworks in addition to Aurora?
>
>
>
> If you only plan to use Aurora for now, I would start with a single Aurora scheduler and see how this works out for you. Aurora does not immediately reject offers but it holds onto them for a default of 5 minutes (IIRC). The single scheduler then has a complete view of all available resources and should be able to schedule your jobs rather quickly.
>
>
>
> Best Regards,
>
> Stephan
>
>
>
> From: Jacob Scott <jacob.scott@gmail.com <ma...@gmail.com>>
> Reply-To: "user@aurora.apache.org <ma...@aurora.apache.org>" <user@aurora.apache.org <ma...@aurora.apache.org>>
> Date: Tuesday 27 September 2016 at 07:34
> To: "user@aurora.apache.org <ma...@aurora.apache.org>" <user@aurora.apache.org <ma...@aurora.apache.org>>
> Subject: roles or attributes for virtual clusters
>
>
>
> I have a mesos cluster running on EC2 that I want to split into two "virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for blue jobs and red nodes/instances for red jobs.
>
>
>
> After reading the documentation I can see two ways of doing this:
>
> Set color:red and color:blue attributes on mesos agents[1] and similarly in Aurora job config [2]
> Using mesos roles [3], Run one blue Aurora scheduler [4] and one red scheduler
> Is one approach here better, or more in line with best practices? Two schedulers seems like more operational overhead, but using attributes might result in a higher offer reject rate (Aurora receiving and rejecting red offers when trying to run a blue job)?
>
>
>
> I appreciate any advice or pointers to relevant resources.
>
>
>
> Thanks,
>
>
>
> Jacob
>
>
>
>
>
> [1] http://mesos.apache.org/documentation/latest/attributes-resources/ <http://mesos.apache.org/documentation/latest/attributes-resources/>
> [2] http://aurora.apache.org/documentation/latest/features/constraints/ <http://aurora.apache.org/documentation/latest/features/constraints/>
> [3] http://mesos.apache.org/documentation/latest/roles/ <http://mesos.apache.org/documentation/latest/roles/>
> [4] http://aurora.apache.org/documentation/latest/reference/scheduler-configuration/ <http://aurora.apache.org/documentation/latest/reference/scheduler-configuration/>
>
Re: roles or attributes for virtual clusters
Posted by Jacob Scott <ja...@gmail.com>.
Thanks much (and thanks for fielding a version of this question in IRC!).
Jacob
On Wed, Sep 28, 2016 at 1:24 PM Zameer Manji <zm...@apache.org> wrote:
> +1
>
> I've found the complexity of managing mesos roles to be high. For a fix
> that has less overhead, I think scheduling on attributes via Aurora's
> constraint system would be easiest.
>
> On Wed, Sep 28, 2016 at 1:20 PM, Jacob Scott <ja...@gmail.com>
> wrote:
>
>> I'm running Chronos and Aurora, with an existing workload which has high
>> reliability requirements. I'm adding in a new, more experimental workload,
>> and would like the different workloads to run on different hardware to
>> reduce risk. Some of the risk comes from me playing fast and loose with
>> per-job cpu/mem/disk, but I expect it to be easier to run for now with
>> different agent classes than untangle that ball of wax. There's also some
>> secondary benefits like better attribution for billing.
>>
>> Based on further reading on roles, and the complexity of reserving
>> resources for roles, I'm currently planning to use attributes -- one
>> Aurora, one Chronos, one Mesos Master, agents classified via attribute).
>>
>>
>> Thanks,
>>
>> Jacob
>>
>> On Wed, Sep 28, 2016 at 1:11 PM, Erb, Stephan <
>> Stephan.Erb@blue-yonder.com> wrote:
>>
>>> Hi Jacob,
>>>
>>>
>>>
>>> could you elaborate why you want to establish two different classes of
>>> agents? Do those have different hardware specs or is this due to a
>>> different network setup or something like that?
>>>
>>>
>>>
>>> Furthermore, do you plan to run other frameworks in addition to Aurora?
>>>
>>>
>>>
>>> If you only plan to use Aurora for now, I would start with a single
>>> Aurora scheduler and see how this works out for you. Aurora does not
>>> immediately reject offers but it holds onto them for a default of 5 minutes
>>> (IIRC). The single scheduler then has a complete view of all available
>>> resources and should be able to schedule your jobs rather quickly.
>>>
>>>
>>>
>>> Best Regards,
>>>
>>> Stephan
>>>
>>>
>>>
>>> *From: *Jacob Scott <ja...@gmail.com>
>>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>> *Date: *Tuesday 27 September 2016 at 07:34
>>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>>> *Subject: *roles or attributes for virtual clusters
>>>
>>>
>>>
>>> I have a mesos cluster running on EC2 that I want to split into two
>>> "virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for
>>> blue jobs and red nodes/instances for red jobs.
>>>
>>>
>>>
>>> After reading the documentation I can see two ways of doing this:
>>>
>>> 1. Set color:red and color:blue attributes on mesos agents[1] and
>>> similarly in Aurora job config [2]
>>> 2. Using mesos roles [3], Run one blue Aurora scheduler [4] and one
>>> red scheduler
>>>
>>> Is one approach here better, or more in line with best practices? Two
>>> schedulers seems like more operational overhead, but using attributes might
>>> result in a higher offer reject rate (Aurora receiving and rejecting red
>>> offers when trying to run a blue job)?
>>>
>>>
>>>
>>> I appreciate any advice or pointers to relevant resources.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>> [1] http://mesos.apache.org/documentation/latest/attributes-resources/
>>>
>>> [2] http://aurora.apache.org/documentation/latest/features/constraints/
>>>
>>> [3] http://mesos.apache.org/documentation/latest/roles/
>>>
>>> [4]
>>> http://aurora.apache.org/documentation/latest/reference/scheduler-configuration/
>>>
>>>
>>
>
Re: roles or attributes for virtual clusters
Posted by Zameer Manji <zm...@apache.org>.
+1
I've found the complexity of managing mesos roles to be high. For a fix
that has less overhead, I think scheduling on attributes via Aurora's
constraint system would be easiest.
On Wed, Sep 28, 2016 at 1:20 PM, Jacob Scott <ja...@gmail.com> wrote:
> I'm running Chronos and Aurora, with an existing workload which has high
> reliability requirements. I'm adding in a new, more experimental workload,
> and would like the different workloads to run on different hardware to
> reduce risk. Some of the risk comes from me playing fast and loose with
> per-job cpu/mem/disk, but I expect it to be easier to run for now with
> different agent classes than untangle that ball of wax. There's also some
> secondary benefits like better attribution for billing.
>
> Based on further reading on roles, and the complexity of reserving
> resources for roles, I'm currently planning to use attributes -- one
> Aurora, one Chronos, one Mesos Master, agents classified via attribute).
>
>
> Thanks,
>
> Jacob
>
> On Wed, Sep 28, 2016 at 1:11 PM, Erb, Stephan <Stephan.Erb@blue-yonder.com
> > wrote:
>
>> Hi Jacob,
>>
>>
>>
>> could you elaborate why you want to establish two different classes of
>> agents? Do those have different hardware specs or is this due to a
>> different network setup or something like that?
>>
>>
>>
>> Furthermore, do you plan to run other frameworks in addition to Aurora?
>>
>>
>>
>> If you only plan to use Aurora for now, I would start with a single
>> Aurora scheduler and see how this works out for you. Aurora does not
>> immediately reject offers but it holds onto them for a default of 5 minutes
>> (IIRC). The single scheduler then has a complete view of all available
>> resources and should be able to schedule your jobs rather quickly.
>>
>>
>>
>> Best Regards,
>>
>> Stephan
>>
>>
>>
>> *From: *Jacob Scott <ja...@gmail.com>
>> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Date: *Tuesday 27 September 2016 at 07:34
>> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
>> *Subject: *roles or attributes for virtual clusters
>>
>>
>>
>> I have a mesos cluster running on EC2 that I want to split into two
>> "virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for
>> blue jobs and red nodes/instances for red jobs.
>>
>>
>>
>> After reading the documentation I can see two ways of doing this:
>>
>> 1. Set color:red and color:blue attributes on mesos agents[1] and
>> similarly in Aurora job config [2]
>> 2. Using mesos roles [3], Run one blue Aurora scheduler [4] and one
>> red scheduler
>>
>> Is one approach here better, or more in line with best practices? Two
>> schedulers seems like more operational overhead, but using attributes might
>> result in a higher offer reject rate (Aurora receiving and rejecting red
>> offers when trying to run a blue job)?
>>
>>
>>
>> I appreciate any advice or pointers to relevant resources.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Jacob
>>
>>
>>
>>
>>
>> [1] http://mesos.apache.org/documentation/latest/attributes-resources/
>>
>> [2] http://aurora.apache.org/documentation/latest/features/constraints/
>>
>> [3] http://mesos.apache.org/documentation/latest/roles/
>>
>> [4] http://aurora.apache.org/documentation/latest/reference/
>> scheduler-configuration/
>>
>>
>
Re: roles or attributes for virtual clusters
Posted by Jacob Scott <ja...@gmail.com>.
I'm running Chronos and Aurora, with an existing workload which has high
reliability requirements. I'm adding in a new, more experimental workload,
and would like the different workloads to run on different hardware to
reduce risk. Some of the risk comes from me playing fast and loose with
per-job cpu/mem/disk, but I expect it to be easier to run for now with
different agent classes than untangle that ball of wax. There's also some
secondary benefits like better attribution for billing.
Based on further reading on roles, and the complexity of reserving
resources for roles, I'm currently planning to use attributes -- one
Aurora, one Chronos, one Mesos Master, agents classified via attribute).
Thanks,
Jacob
On Wed, Sep 28, 2016 at 1:11 PM, Erb, Stephan <St...@blue-yonder.com>
wrote:
> Hi Jacob,
>
>
>
> could you elaborate why you want to establish two different classes of
> agents? Do those have different hardware specs or is this due to a
> different network setup or something like that?
>
>
>
> Furthermore, do you plan to run other frameworks in addition to Aurora?
>
>
>
> If you only plan to use Aurora for now, I would start with a single Aurora
> scheduler and see how this works out for you. Aurora does not immediately
> reject offers but it holds onto them for a default of 5 minutes (IIRC). The
> single scheduler then has a complete view of all available resources and
> should be able to schedule your jobs rather quickly.
>
>
>
> Best Regards,
>
> Stephan
>
>
>
> *From: *Jacob Scott <ja...@gmail.com>
> *Reply-To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Date: *Tuesday 27 September 2016 at 07:34
> *To: *"user@aurora.apache.org" <us...@aurora.apache.org>
> *Subject: *roles or attributes for virtual clusters
>
>
>
> I have a mesos cluster running on EC2 that I want to split into two
> "virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for
> blue jobs and red nodes/instances for red jobs.
>
>
>
> After reading the documentation I can see two ways of doing this:
>
> 1. Set color:red and color:blue attributes on mesos agents[1] and
> similarly in Aurora job config [2]
> 2. Using mesos roles [3], Run one blue Aurora scheduler [4] and one
> red scheduler
>
> Is one approach here better, or more in line with best practices? Two
> schedulers seems like more operational overhead, but using attributes might
> result in a higher offer reject rate (Aurora receiving and rejecting red
> offers when trying to run a blue job)?
>
>
>
> I appreciate any advice or pointers to relevant resources.
>
>
>
> Thanks,
>
>
>
> Jacob
>
>
>
>
>
> [1] http://mesos.apache.org/documentation/latest/attributes-resources/
>
> [2] http://aurora.apache.org/documentation/latest/features/constraints/
>
> [3] http://mesos.apache.org/documentation/latest/roles/
>
> [4] http://aurora.apache.org/documentation/latest/reference/scheduler-
> configuration/
>
>
Re: roles or attributes for virtual clusters
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
Hi Jacob,
could you elaborate why you want to establish two different classes of agents? Do those have different hardware specs or is this due to a different network setup or something like that?
Furthermore, do you plan to run other frameworks in addition to Aurora?
If you only plan to use Aurora for now, I would start with a single Aurora scheduler and see how this works out for you. Aurora does not immediately reject offers but it holds onto them for a default of 5 minutes (IIRC). The single scheduler then has a complete view of all available resources and should be able to schedule your jobs rather quickly.
Best Regards,
Stephan
From: Jacob Scott <ja...@gmail.com>
Reply-To: "user@aurora.apache.org" <us...@aurora.apache.org>
Date: Tuesday 27 September 2016 at 07:34
To: "user@aurora.apache.org" <us...@aurora.apache.org>
Subject: roles or attributes for virtual clusters
I have a mesos cluster running on EC2 that I want to split into two "virtual clusters" to run Aurora jobs on -- e.g. blue nodes/instances for blue jobs and red nodes/instances for red jobs.
After reading the documentation I can see two ways of doing this:
1. Set color:red and color:blue attributes on mesos agents[1] and similarly in Aurora job config [2]
2. Using mesos roles [3], Run one blue Aurora scheduler [4] and one red scheduler
Is one approach here better, or more in line with best practices? Two schedulers seems like more operational overhead, but using attributes might result in a higher offer reject rate (Aurora receiving and rejecting red offers when trying to run a blue job)?
I appreciate any advice or pointers to relevant resources.
Thanks,
Jacob
[1] http://mesos.apache.org/documentation/latest/attributes-resources/
[2] http://aurora.apache.org/documentation/latest/features/constraints/
[3] http://mesos.apache.org/documentation/latest/roles/
[4] http://aurora.apache.org/documentation/latest/reference/scheduler-configuration/