You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Nihal Harish <ni...@gmail.com> on 2016/05/31 06:33:48 UTC
Fwd: Request offers based on Slave Attributes
Hi,
The mesos.proto file allows us to specify request offers based on only
slave_id and resources, is there some way to request offers based on slave
attributes and/or hostname?
Thanks.
Regards,
Nihal
Re: Request offers based on Slave Attributes
Posted by Nihal Harish <ni...@gmail.com>.
The issue I face occurs under the following circumstance:
The allocator initially provides five offers, out of which my framework,
picks one and executes a task.
The allocator then, begins to provide the offer that was selected
previously by the framework.
Is there any way to force the allocator to offer the same resources for
every request?
On 31 May 2016 at 14:19, <co...@gmail.com> wrote:
> Replacing the allocator is possible but probably not the best path
> forward. It's a big task to write your own. The requests are only meant to
> be hints for the allocator anyway. Is offer latency a bottleneck for you?
> With static or dynamic reservations you can earmark slabs of resources for
> various framework roles (
> http://mesos.apache.org/documentation/latest/reservation/). You might
> also try tuning the master knobs --allocation_interval and
> --offer-timeout to mitigate against frameworks that hoard offers (e.g.
> inconsiderate or buggy schedulers).
>
> On May 31, 2016, at 01:31, Nihal Harish <ni...@gmail.com> wrote:
>
> So the best approach would be to write my own hierarchical allocator? As
> of now, I am currently filtering out offers made, but this adds latency
> because, the required resources may not be offered for every request.
>
> On 31 May 2016 at 13:34, <co...@gmail.com> wrote:
>
>> Just FYI, note the default hierarchical allocator doesn't do anything
>> with the resource requests it receives. See
>> https://github.com/apache/mesos/blob/master/src/master/allocator/mesos/hierarchical.cpp
>>
>> On May 30, 2016, at 23:58, Guangya Liu <gy...@gmail.com> wrote:
>>
>> I think that you can filter the offers based on slave attribute, all of
>> the offers are assigned agent attributes in the offer:
>> https://github.com/apache/mesos/blob/master/src/master/master.cpp#L5637
>>
>> On Tue, May 31, 2016 at 2:33 PM, Nihal Harish <ni...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The mesos.proto file allows us to specify request offers based on only
>>> slave_id and resources, is there some way to request offers based on slave
>>> attributes and/or hostname?
>>>
>>> Thanks.
>>> Regards,
>>> Nihal
>>>
>>>
>>
>
Re: Request offers based on Slave Attributes
Posted by co...@gmail.com.
Replacing the allocator is possible but probably not the best path forward. It's a big task to write your own. The requests are only meant to be hints for the allocator anyway. Is offer latency a bottleneck for you? With static or dynamic reservations you can earmark slabs of resources for various framework roles (http://mesos.apache.org/documentation/latest/reservation/). You might also try tuning the master knobs --allocation_interval and --offer-timeout to mitigate against frameworks that hoard offers (e.g. inconsiderate or buggy schedulers).
> On May 31, 2016, at 01:31, Nihal Harish <ni...@gmail.com> wrote:
>
> So the best approach would be to write my own hierarchical allocator? As of now, I am currently filtering out offers made, but this adds latency because, the required resources may not be offered for every request.
>
>> On 31 May 2016 at 13:34, <co...@gmail.com> wrote:
>> Just FYI, note the default hierarchical allocator doesn't do anything with the resource requests it receives. See https://github.com/apache/mesos/blob/master/src/master/allocator/mesos/hierarchical.cpp
>>
>> On May 30, 2016, at 23:58, Guangya Liu <gy...@gmail.com> wrote:
>>
>>> I think that you can filter the offers based on slave attribute, all of the offers are assigned agent attributes in the offer: https://github.com/apache/mesos/blob/master/src/master/master.cpp#L5637
>>>
>>>> On Tue, May 31, 2016 at 2:33 PM, Nihal Harish <ni...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> The mesos.proto file allows us to specify request offers based on only slave_id and resources, is there some way to request offers based on slave attributes and/or hostname?
>>>>
>>>> Thanks.
>>>> Regards,
>>>> Nihal
>
Re: Request offers based on Slave Attributes
Posted by Nihal Harish <ni...@gmail.com>.
So the best approach would be to write my own hierarchical allocator? As of
now, I am currently filtering out offers made, but this adds latency
because, the required resources may not be offered for every request.
On 31 May 2016 at 13:34, <co...@gmail.com> wrote:
> Just FYI, note the default hierarchical allocator doesn't do anything with
> the resource requests it receives. See
> https://github.com/apache/mesos/blob/master/src/master/allocator/mesos/hierarchical.cpp
>
> On May 30, 2016, at 23:58, Guangya Liu <gy...@gmail.com> wrote:
>
> I think that you can filter the offers based on slave attribute, all of
> the offers are assigned agent attributes in the offer:
> https://github.com/apache/mesos/blob/master/src/master/master.cpp#L5637
>
> On Tue, May 31, 2016 at 2:33 PM, Nihal Harish <ni...@gmail.com>
> wrote:
>
>> Hi,
>>
>> The mesos.proto file allows us to specify request offers based on only
>> slave_id and resources, is there some way to request offers based on slave
>> attributes and/or hostname?
>>
>> Thanks.
>> Regards,
>> Nihal
>>
>>
>
Re: Request offers based on Slave Attributes
Posted by co...@gmail.com.
Just FYI, note the default hierarchical allocator doesn't do anything with the resource requests it receives. See https://github.com/apache/mesos/blob/master/src/master/allocator/mesos/hierarchical.cpp
> On May 30, 2016, at 23:58, Guangya Liu <gy...@gmail.com> wrote:
>
> I think that you can filter the offers based on slave attribute, all of the offers are assigned agent attributes in the offer: https://github.com/apache/mesos/blob/master/src/master/master.cpp#L5637
>
>> On Tue, May 31, 2016 at 2:33 PM, Nihal Harish <ni...@gmail.com> wrote:
>> Hi,
>>
>> The mesos.proto file allows us to specify request offers based on only slave_id and resources, is there some way to request offers based on slave attributes and/or hostname?
>>
>> Thanks.
>> Regards,
>> Nihal
>
RE: Request offers based on Slave Attributes
Posted by Klaus Ma <kl...@gmail.com>.
Hi Nihal,
Currently, the Mesos master/allocator will ignore "requestResources"; as Guangya said, please filter out the necessary by slave's attributes when you got offers.
----Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer Platform OpenSource Technology, STG, IBM GCG +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
Date: Tue, 31 May 2016 14:58:39 +0800
Subject: Re: Request offers based on Slave Attributes
From: gyliu513@gmail.com
To: user@mesos.apache.org
I think that you can filter the offers based on slave attribute, all of the offers are assigned agent attributes in the offer: https://github.com/apache/mesos/blob/master/src/master/master.cpp#L5637
On Tue, May 31, 2016 at 2:33 PM, Nihal Harish <ni...@gmail.com> wrote:
Hi,The mesos.proto file allows us to specify request offers based on only slave_id and resources, is there some way to request offers based on slave attributes and/or hostname?Thanks.Regards,Nihal
Re: Request offers based on Slave Attributes
Posted by Guangya Liu <gy...@gmail.com>.
I think that you can filter the offers based on slave attribute, all of the
offers are assigned agent attributes in the offer:
https://github.com/apache/mesos/blob/master/src/master/master.cpp#L5637
On Tue, May 31, 2016 at 2:33 PM, Nihal Harish <ni...@gmail.com>
wrote:
> Hi,
>
> The mesos.proto file allows us to specify request offers based on only
> slave_id and resources, is there some way to request offers based on slave
> attributes and/or hostname?
>
> Thanks.
> Regards,
> Nihal
>
>