You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Meng Zhu <mz...@mesosphere.com> on 2018/06/11 22:47:01 UTC
Proposing change to the allocatable check in the allocator
Hi:
The allocatable
<https://github.com/apache/mesos/blob/1.5.x/src/master/allocator/mesos/hierarchical.cpp#L2471-L2479>
check in the allocator (shown below) was originally introduced to
help alleviate the situation where a framework receives some resources, but
no
cpu/memory, thus cannot launch a task.
constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM = Megabytes(32);
bool HierarchicalAllocatorProcess::allocatable(
const Resources& resources)
{
Option<double> cpus = resources.cpus();
Option<Bytes> mem = resources.mem();
return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
(mem.isSome() && mem.get() >= MIN_MEM);
}
Issues
However, there has been a couple of issues surfacing lately surrounding the
check.
-
- - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
memory-only offers.
We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
When we
allocate resources to a role, we'll "chop" the available resources of the
agent up to the
quota limit for the role. However, this has the unintended consequence of
creating
cpu-only and memory-only offers, even though there might be other agents
with both
cpu and memory resources available in the cluster.
- MESOS-8626 The 'allocatable' check in the allocator is problematic with
multi-role frameworks.
Consider roleA reserved cpu/memory on an agent and roleB reserved disk on
the same agent.
A framework under both roleA and roleB will not be able to get the reserved
disk due to the
allocatable check. With the introduction of resource providers, the similar
situation will
become more common.
Proposed change
Instead of hardcoding a one-size-fits-all value in Mesos, we are proposing
to add a new master flag
min_allocatable_resources. It specifies one or more scalar resources
quantities that define the
minimum allocatable resources for the allocator. The allocator will only
offer resources that are more
than at least one of the specified resources. The default behavior *is
backward compatible* i.e.
by default, the flag is set to “cpus:0.01|mem:32”.
Usage
The flag takes in either a simple text of resource(s) delimited by a bar
(|) or a JSON array of JSON
formatted resources. Note, the input should be “pure” scalar quantities
i.e. the specified resource(s)
should only have name, type (set to scalar) and scalar fields set.
Examples:
- - To eliminate cpu or memory only offer due to the quota chopping,
- we could set the flag to “cpus:0.01;mem:32”
-
- - To enable offering disk only offer, we could set the flag to
“disk:32”
-
- - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
- Then the allocator will only offer resources that at least contain
“cpus:0.01;mem:32”
- OR resources that at least contain “disk:32”.
Let me know what you think! Thanks!
-Meng
Re: Proposing change to the allocatable check in the allocator
Posted by Greg Mann <gr...@mesosphere.io>.
Hi all,
We had a nice discussion about this in the API working group meeting today.
I agree that it's a good idea to do our best to make this change compatible
with future updates to the Request call and/or quota. I think it would be
beneficial to have a meeting in a few days to brainstorm some ideas; please
let me know if you would like to be included in that meeting and I will add
you to an invite!
Cheers,
Greg
On Tue, Jun 12, 2018 at 8:06 AM, Alex Rukletsov <al...@mesosphere.com> wrote:
> Instead of the master flag, why not a master API call. This will allow to
> update the value without restarting the master.
>
> Another thought is that we should explain operators how and when to use
> this knob. For example, if they observe a behavioural pattern A, then it
> means B is happening, and tuning the knob to C might help.
>
> On Tue, Jun 12, 2018 at 7:36 AM, Jie Yu <yu...@gmail.com> wrote:
>
>> I would suggest we also consider the possibility of adding per framework
>> control on `min_allocatable_resources`.
>>
>> If we want to consider supporting per-framework setting, we should
>> probably
>> model this as a protobuf, rather than a free form JSON. The same protobuf
>> can be reused for both master flag, framework API, or even supporting
>> Resource Request in the future. Something like the following:
>>
>> message ResourceQuantityPredicate {
>> enum Type {
>> SCALAR_GE,
>> }
>> optional Type type;
>> optional Value.Scalar scalar;
>> }
>> message ResourceRequirement {
>> required string resource_name;
>> oneof predicates {
>> ResourceQuantityPredicate quantity;
>> }
>> }
>> message ResourceRequirementList {
>> // All requirements MUST be met.
>> repeated ResourceRequirement requirements;
>> }
>>
>> // Resource request API.
>> message Request {
>> repeated ResoruceRequrementList accepted;
>> }
>>
>> // `allocatable()`
>> message MinimalAllocatableResources {
>> repeated ResoruceRequrementList accepted;
>> }
>>
>> On Mon, Jun 11, 2018 at 3:47 PM, Meng Zhu <mz...@mesosphere.com> wrote:
>>
>> > Hi:
>> >
>> > The allocatable
>> > <https://github.com/apache/mesos/blob/1.5.x/src/master/alloc
>> ator/mesos/hierarchical.cpp#L2471-L2479>
>> > check in the allocator (shown below) was originally introduced to
>> >
>> > help alleviate the situation where a framework receives some resources,
>> > but no
>> >
>> > cpu/memory, thus cannot launch a task.
>> >
>> >
>> > constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM =
>> Megabytes(32);
>> > bool HierarchicalAllocatorProcess::allocatable(
>> > const Resources& resources)
>> > {
>> > Option<double> cpus = resources.cpus();
>> > Option<Bytes> mem = resources.mem();
>> >
>> > return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
>> > (mem.isSome() && mem.get() >= MIN_MEM);
>> > }
>> >
>> >
>> > Issues
>> >
>> > However, there has been a couple of issues surfacing lately surrounding
>> > the check.
>> >
>> > -
>> > - - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
>>
>> > memory-only offers.
>> >
>> > We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
>> > When we
>> >
>> > allocate resources to a role, we'll "chop" the available resources of
>> the
>> > agent up to the
>> >
>> > quota limit for the role. However, this has the unintended consequence
>> of
>> > creating
>> >
>> > cpu-only and memory-only offers, even though there might be other agents
>> > with both
>> >
>> > cpu and memory resources available in the cluster.
>> >
>> >
>> > - MESOS-8626 The 'allocatable' check in the allocator is problematic
>> with
>> > multi-role frameworks.
>> >
>> > Consider roleA reserved cpu/memory on an agent and roleB reserved disk
>> on
>> > the same agent.
>> >
>> > A framework under both roleA and roleB will not be able to get the
>> > reserved disk due to the
>> >
>> > allocatable check. With the introduction of resource providers, the
>> > similar situation will
>> >
>> > become more common.
>> >
>> > Proposed change
>> >
>> > Instead of hardcoding a one-size-fits-all value in Mesos, we are
>> proposing
>> > to add a new master flag
>> >
>> > min_allocatable_resources. It specifies one or more scalar resources
>> > quantities that define the
>> >
>> > minimum allocatable resources for the allocator. The allocator will only
>> > offer resources that are more
>> >
>> > than at least one of the specified resources. The default behavior *is
>> > backward compatible* i.e.
>> >
>> > by default, the flag is set to “cpus:0.01|mem:32”.
>> >
>> > Usage
>> >
>> > The flag takes in either a simple text of resource(s) delimited by a bar
>> > (|) or a JSON array of JSON
>> >
>> > formatted resources. Note, the input should be “pure” scalar quantities
>> > i.e. the specified resource(s)
>> >
>> > should only have name, type (set to scalar) and scalar fields set.
>> >
>> >
>> > Examples:
>> >
>> > - - To eliminate cpu or memory only offer due to the quota chopping,
>> > - we could set the flag to “cpus:0.01;mem:32”
>> > -
>> > - - To enable offering disk only offer, we could set the flag to
>> > “disk:32”
>> > -
>> > - - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
>> > - Then the allocator will only offer resources that at least contain
>> > “cpus:0.01;mem:32”
>> > - OR resources that at least contain “disk:32”.
>> >
>> >
>> > Let me know what you think! Thanks!
>> >
>> >
>> > -Meng
>> >
>> >
>>
>
>
Re: Proposing change to the allocatable check in the allocator
Posted by Greg Mann <gr...@mesosphere.io>.
Hi all,
We had a nice discussion about this in the API working group meeting today.
I agree that it's a good idea to do our best to make this change compatible
with future updates to the Request call and/or quota. I think it would be
beneficial to have a meeting in a few days to brainstorm some ideas; please
let me know if you would like to be included in that meeting and I will add
you to an invite!
Cheers,
Greg
On Tue, Jun 12, 2018 at 8:06 AM, Alex Rukletsov <al...@mesosphere.com> wrote:
> Instead of the master flag, why not a master API call. This will allow to
> update the value without restarting the master.
>
> Another thought is that we should explain operators how and when to use
> this knob. For example, if they observe a behavioural pattern A, then it
> means B is happening, and tuning the knob to C might help.
>
> On Tue, Jun 12, 2018 at 7:36 AM, Jie Yu <yu...@gmail.com> wrote:
>
>> I would suggest we also consider the possibility of adding per framework
>> control on `min_allocatable_resources`.
>>
>> If we want to consider supporting per-framework setting, we should
>> probably
>> model this as a protobuf, rather than a free form JSON. The same protobuf
>> can be reused for both master flag, framework API, or even supporting
>> Resource Request in the future. Something like the following:
>>
>> message ResourceQuantityPredicate {
>> enum Type {
>> SCALAR_GE,
>> }
>> optional Type type;
>> optional Value.Scalar scalar;
>> }
>> message ResourceRequirement {
>> required string resource_name;
>> oneof predicates {
>> ResourceQuantityPredicate quantity;
>> }
>> }
>> message ResourceRequirementList {
>> // All requirements MUST be met.
>> repeated ResourceRequirement requirements;
>> }
>>
>> // Resource request API.
>> message Request {
>> repeated ResoruceRequrementList accepted;
>> }
>>
>> // `allocatable()`
>> message MinimalAllocatableResources {
>> repeated ResoruceRequrementList accepted;
>> }
>>
>> On Mon, Jun 11, 2018 at 3:47 PM, Meng Zhu <mz...@mesosphere.com> wrote:
>>
>> > Hi:
>> >
>> > The allocatable
>> > <https://github.com/apache/mesos/blob/1.5.x/src/master/alloc
>> ator/mesos/hierarchical.cpp#L2471-L2479>
>> > check in the allocator (shown below) was originally introduced to
>> >
>> > help alleviate the situation where a framework receives some resources,
>> > but no
>> >
>> > cpu/memory, thus cannot launch a task.
>> >
>> >
>> > constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM =
>> Megabytes(32);
>> > bool HierarchicalAllocatorProcess::allocatable(
>> > const Resources& resources)
>> > {
>> > Option<double> cpus = resources.cpus();
>> > Option<Bytes> mem = resources.mem();
>> >
>> > return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
>> > (mem.isSome() && mem.get() >= MIN_MEM);
>> > }
>> >
>> >
>> > Issues
>> >
>> > However, there has been a couple of issues surfacing lately surrounding
>> > the check.
>> >
>> > -
>> > - - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
>>
>> > memory-only offers.
>> >
>> > We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
>> > When we
>> >
>> > allocate resources to a role, we'll "chop" the available resources of
>> the
>> > agent up to the
>> >
>> > quota limit for the role. However, this has the unintended consequence
>> of
>> > creating
>> >
>> > cpu-only and memory-only offers, even though there might be other agents
>> > with both
>> >
>> > cpu and memory resources available in the cluster.
>> >
>> >
>> > - MESOS-8626 The 'allocatable' check in the allocator is problematic
>> with
>> > multi-role frameworks.
>> >
>> > Consider roleA reserved cpu/memory on an agent and roleB reserved disk
>> on
>> > the same agent.
>> >
>> > A framework under both roleA and roleB will not be able to get the
>> > reserved disk due to the
>> >
>> > allocatable check. With the introduction of resource providers, the
>> > similar situation will
>> >
>> > become more common.
>> >
>> > Proposed change
>> >
>> > Instead of hardcoding a one-size-fits-all value in Mesos, we are
>> proposing
>> > to add a new master flag
>> >
>> > min_allocatable_resources. It specifies one or more scalar resources
>> > quantities that define the
>> >
>> > minimum allocatable resources for the allocator. The allocator will only
>> > offer resources that are more
>> >
>> > than at least one of the specified resources. The default behavior *is
>> > backward compatible* i.e.
>> >
>> > by default, the flag is set to “cpus:0.01|mem:32”.
>> >
>> > Usage
>> >
>> > The flag takes in either a simple text of resource(s) delimited by a bar
>> > (|) or a JSON array of JSON
>> >
>> > formatted resources. Note, the input should be “pure” scalar quantities
>> > i.e. the specified resource(s)
>> >
>> > should only have name, type (set to scalar) and scalar fields set.
>> >
>> >
>> > Examples:
>> >
>> > - - To eliminate cpu or memory only offer due to the quota chopping,
>> > - we could set the flag to “cpus:0.01;mem:32”
>> > -
>> > - - To enable offering disk only offer, we could set the flag to
>> > “disk:32”
>> > -
>> > - - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
>> > - Then the allocator will only offer resources that at least contain
>> > “cpus:0.01;mem:32”
>> > - OR resources that at least contain “disk:32”.
>> >
>> >
>> > Let me know what you think! Thanks!
>> >
>> >
>> > -Meng
>> >
>> >
>>
>
>
Re: Proposing change to the allocatable check in the allocator
Posted by Alex Rukletsov <al...@mesosphere.com>.
Instead of the master flag, why not a master API call. This will allow to
update the value without restarting the master.
Another thought is that we should explain operators how and when to use
this knob. For example, if they observe a behavioural pattern A, then it
means B is happening, and tuning the knob to C might help.
On Tue, Jun 12, 2018 at 7:36 AM, Jie Yu <yu...@gmail.com> wrote:
> I would suggest we also consider the possibility of adding per framework
> control on `min_allocatable_resources`.
>
> If we want to consider supporting per-framework setting, we should probably
> model this as a protobuf, rather than a free form JSON. The same protobuf
> can be reused for both master flag, framework API, or even supporting
> Resource Request in the future. Something like the following:
>
> message ResourceQuantityPredicate {
> enum Type {
> SCALAR_GE,
> }
> optional Type type;
> optional Value.Scalar scalar;
> }
> message ResourceRequirement {
> required string resource_name;
> oneof predicates {
> ResourceQuantityPredicate quantity;
> }
> }
> message ResourceRequirementList {
> // All requirements MUST be met.
> repeated ResourceRequirement requirements;
> }
>
> // Resource request API.
> message Request {
> repeated ResoruceRequrementList accepted;
> }
>
> // `allocatable()`
> message MinimalAllocatableResources {
> repeated ResoruceRequrementList accepted;
> }
>
> On Mon, Jun 11, 2018 at 3:47 PM, Meng Zhu <mz...@mesosphere.com> wrote:
>
> > Hi:
> >
> > The allocatable
> > <https://github.com/apache/mesos/blob/1.5.x/src/master/
> allocator/mesos/hierarchical.cpp#L2471-L2479>
> > check in the allocator (shown below) was originally introduced to
> >
> > help alleviate the situation where a framework receives some resources,
> > but no
> >
> > cpu/memory, thus cannot launch a task.
> >
> >
> > constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM = Megabytes(32);
> > bool HierarchicalAllocatorProcess::allocatable(
> > const Resources& resources)
> > {
> > Option<double> cpus = resources.cpus();
> > Option<Bytes> mem = resources.mem();
> >
> > return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
> > (mem.isSome() && mem.get() >= MIN_MEM);
> > }
> >
> >
> > Issues
> >
> > However, there has been a couple of issues surfacing lately surrounding
> > the check.
> >
> > -
> > - - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
> > memory-only offers.
> >
> > We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
> > When we
> >
> > allocate resources to a role, we'll "chop" the available resources of the
> > agent up to the
> >
> > quota limit for the role. However, this has the unintended consequence of
> > creating
> >
> > cpu-only and memory-only offers, even though there might be other agents
> > with both
> >
> > cpu and memory resources available in the cluster.
> >
> >
> > - MESOS-8626 The 'allocatable' check in the allocator is problematic with
> > multi-role frameworks.
> >
> > Consider roleA reserved cpu/memory on an agent and roleB reserved disk on
> > the same agent.
> >
> > A framework under both roleA and roleB will not be able to get the
> > reserved disk due to the
> >
> > allocatable check. With the introduction of resource providers, the
> > similar situation will
> >
> > become more common.
> >
> > Proposed change
> >
> > Instead of hardcoding a one-size-fits-all value in Mesos, we are
> proposing
> > to add a new master flag
> >
> > min_allocatable_resources. It specifies one or more scalar resources
> > quantities that define the
> >
> > minimum allocatable resources for the allocator. The allocator will only
> > offer resources that are more
> >
> > than at least one of the specified resources. The default behavior *is
> > backward compatible* i.e.
> >
> > by default, the flag is set to “cpus:0.01|mem:32”.
> >
> > Usage
> >
> > The flag takes in either a simple text of resource(s) delimited by a bar
> > (|) or a JSON array of JSON
> >
> > formatted resources. Note, the input should be “pure” scalar quantities
> > i.e. the specified resource(s)
> >
> > should only have name, type (set to scalar) and scalar fields set.
> >
> >
> > Examples:
> >
> > - - To eliminate cpu or memory only offer due to the quota chopping,
> > - we could set the flag to “cpus:0.01;mem:32”
> > -
> > - - To enable offering disk only offer, we could set the flag to
> > “disk:32”
> > -
> > - - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
> > - Then the allocator will only offer resources that at least contain
> > “cpus:0.01;mem:32”
> > - OR resources that at least contain “disk:32”.
> >
> >
> > Let me know what you think! Thanks!
> >
> >
> > -Meng
> >
> >
>
Re: Proposing change to the allocatable check in the allocator
Posted by Alex Rukletsov <al...@mesosphere.com>.
Instead of the master flag, why not a master API call. This will allow to
update the value without restarting the master.
Another thought is that we should explain operators how and when to use
this knob. For example, if they observe a behavioural pattern A, then it
means B is happening, and tuning the knob to C might help.
On Tue, Jun 12, 2018 at 7:36 AM, Jie Yu <yu...@gmail.com> wrote:
> I would suggest we also consider the possibility of adding per framework
> control on `min_allocatable_resources`.
>
> If we want to consider supporting per-framework setting, we should probably
> model this as a protobuf, rather than a free form JSON. The same protobuf
> can be reused for both master flag, framework API, or even supporting
> Resource Request in the future. Something like the following:
>
> message ResourceQuantityPredicate {
> enum Type {
> SCALAR_GE,
> }
> optional Type type;
> optional Value.Scalar scalar;
> }
> message ResourceRequirement {
> required string resource_name;
> oneof predicates {
> ResourceQuantityPredicate quantity;
> }
> }
> message ResourceRequirementList {
> // All requirements MUST be met.
> repeated ResourceRequirement requirements;
> }
>
> // Resource request API.
> message Request {
> repeated ResoruceRequrementList accepted;
> }
>
> // `allocatable()`
> message MinimalAllocatableResources {
> repeated ResoruceRequrementList accepted;
> }
>
> On Mon, Jun 11, 2018 at 3:47 PM, Meng Zhu <mz...@mesosphere.com> wrote:
>
> > Hi:
> >
> > The allocatable
> > <https://github.com/apache/mesos/blob/1.5.x/src/master/
> allocator/mesos/hierarchical.cpp#L2471-L2479>
> > check in the allocator (shown below) was originally introduced to
> >
> > help alleviate the situation where a framework receives some resources,
> > but no
> >
> > cpu/memory, thus cannot launch a task.
> >
> >
> > constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM = Megabytes(32);
> > bool HierarchicalAllocatorProcess::allocatable(
> > const Resources& resources)
> > {
> > Option<double> cpus = resources.cpus();
> > Option<Bytes> mem = resources.mem();
> >
> > return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
> > (mem.isSome() && mem.get() >= MIN_MEM);
> > }
> >
> >
> > Issues
> >
> > However, there has been a couple of issues surfacing lately surrounding
> > the check.
> >
> > -
> > - - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
> > memory-only offers.
> >
> > We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
> > When we
> >
> > allocate resources to a role, we'll "chop" the available resources of the
> > agent up to the
> >
> > quota limit for the role. However, this has the unintended consequence of
> > creating
> >
> > cpu-only and memory-only offers, even though there might be other agents
> > with both
> >
> > cpu and memory resources available in the cluster.
> >
> >
> > - MESOS-8626 The 'allocatable' check in the allocator is problematic with
> > multi-role frameworks.
> >
> > Consider roleA reserved cpu/memory on an agent and roleB reserved disk on
> > the same agent.
> >
> > A framework under both roleA and roleB will not be able to get the
> > reserved disk due to the
> >
> > allocatable check. With the introduction of resource providers, the
> > similar situation will
> >
> > become more common.
> >
> > Proposed change
> >
> > Instead of hardcoding a one-size-fits-all value in Mesos, we are
> proposing
> > to add a new master flag
> >
> > min_allocatable_resources. It specifies one or more scalar resources
> > quantities that define the
> >
> > minimum allocatable resources for the allocator. The allocator will only
> > offer resources that are more
> >
> > than at least one of the specified resources. The default behavior *is
> > backward compatible* i.e.
> >
> > by default, the flag is set to “cpus:0.01|mem:32”.
> >
> > Usage
> >
> > The flag takes in either a simple text of resource(s) delimited by a bar
> > (|) or a JSON array of JSON
> >
> > formatted resources. Note, the input should be “pure” scalar quantities
> > i.e. the specified resource(s)
> >
> > should only have name, type (set to scalar) and scalar fields set.
> >
> >
> > Examples:
> >
> > - - To eliminate cpu or memory only offer due to the quota chopping,
> > - we could set the flag to “cpus:0.01;mem:32”
> > -
> > - - To enable offering disk only offer, we could set the flag to
> > “disk:32”
> > -
> > - - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
> > - Then the allocator will only offer resources that at least contain
> > “cpus:0.01;mem:32”
> > - OR resources that at least contain “disk:32”.
> >
> >
> > Let me know what you think! Thanks!
> >
> >
> > -Meng
> >
> >
>
Re: Proposing change to the allocatable check in the allocator
Posted by Jie Yu <yu...@gmail.com>.
I would suggest we also consider the possibility of adding per framework
control on `min_allocatable_resources`.
If we want to consider supporting per-framework setting, we should probably
model this as a protobuf, rather than a free form JSON. The same protobuf
can be reused for both master flag, framework API, or even supporting
Resource Request in the future. Something like the following:
message ResourceQuantityPredicate {
enum Type {
SCALAR_GE,
}
optional Type type;
optional Value.Scalar scalar;
}
message ResourceRequirement {
required string resource_name;
oneof predicates {
ResourceQuantityPredicate quantity;
}
}
message ResourceRequirementList {
// All requirements MUST be met.
repeated ResourceRequirement requirements;
}
// Resource request API.
message Request {
repeated ResoruceRequrementList accepted;
}
// `allocatable()`
message MinimalAllocatableResources {
repeated ResoruceRequrementList accepted;
}
On Mon, Jun 11, 2018 at 3:47 PM, Meng Zhu <mz...@mesosphere.com> wrote:
> Hi:
>
> The allocatable
> <https://github.com/apache/mesos/blob/1.5.x/src/master/allocator/mesos/hierarchical.cpp#L2471-L2479>
> check in the allocator (shown below) was originally introduced to
>
> help alleviate the situation where a framework receives some resources,
> but no
>
> cpu/memory, thus cannot launch a task.
>
>
> constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM = Megabytes(32);
> bool HierarchicalAllocatorProcess::allocatable(
> const Resources& resources)
> {
> Option<double> cpus = resources.cpus();
> Option<Bytes> mem = resources.mem();
>
> return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
> (mem.isSome() && mem.get() >= MIN_MEM);
> }
>
>
> Issues
>
> However, there has been a couple of issues surfacing lately surrounding
> the check.
>
> -
> - - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
> memory-only offers.
>
> We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
> When we
>
> allocate resources to a role, we'll "chop" the available resources of the
> agent up to the
>
> quota limit for the role. However, this has the unintended consequence of
> creating
>
> cpu-only and memory-only offers, even though there might be other agents
> with both
>
> cpu and memory resources available in the cluster.
>
>
> - MESOS-8626 The 'allocatable' check in the allocator is problematic with
> multi-role frameworks.
>
> Consider roleA reserved cpu/memory on an agent and roleB reserved disk on
> the same agent.
>
> A framework under both roleA and roleB will not be able to get the
> reserved disk due to the
>
> allocatable check. With the introduction of resource providers, the
> similar situation will
>
> become more common.
>
> Proposed change
>
> Instead of hardcoding a one-size-fits-all value in Mesos, we are proposing
> to add a new master flag
>
> min_allocatable_resources. It specifies one or more scalar resources
> quantities that define the
>
> minimum allocatable resources for the allocator. The allocator will only
> offer resources that are more
>
> than at least one of the specified resources. The default behavior *is
> backward compatible* i.e.
>
> by default, the flag is set to “cpus:0.01|mem:32”.
>
> Usage
>
> The flag takes in either a simple text of resource(s) delimited by a bar
> (|) or a JSON array of JSON
>
> formatted resources. Note, the input should be “pure” scalar quantities
> i.e. the specified resource(s)
>
> should only have name, type (set to scalar) and scalar fields set.
>
>
> Examples:
>
> - - To eliminate cpu or memory only offer due to the quota chopping,
> - we could set the flag to “cpus:0.01;mem:32”
> -
> - - To enable offering disk only offer, we could set the flag to
> “disk:32”
> -
> - - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
> - Then the allocator will only offer resources that at least contain
> “cpus:0.01;mem:32”
> - OR resources that at least contain “disk:32”.
>
>
> Let me know what you think! Thanks!
>
>
> -Meng
>
>
Re: Proposing change to the allocatable check in the allocator
Posted by Jie Yu <yu...@gmail.com>.
I would suggest we also consider the possibility of adding per framework
control on `min_allocatable_resources`.
If we want to consider supporting per-framework setting, we should probably
model this as a protobuf, rather than a free form JSON. The same protobuf
can be reused for both master flag, framework API, or even supporting
Resource Request in the future. Something like the following:
message ResourceQuantityPredicate {
enum Type {
SCALAR_GE,
}
optional Type type;
optional Value.Scalar scalar;
}
message ResourceRequirement {
required string resource_name;
oneof predicates {
ResourceQuantityPredicate quantity;
}
}
message ResourceRequirementList {
// All requirements MUST be met.
repeated ResourceRequirement requirements;
}
// Resource request API.
message Request {
repeated ResoruceRequrementList accepted;
}
// `allocatable()`
message MinimalAllocatableResources {
repeated ResoruceRequrementList accepted;
}
On Mon, Jun 11, 2018 at 3:47 PM, Meng Zhu <mz...@mesosphere.com> wrote:
> Hi:
>
> The allocatable
> <https://github.com/apache/mesos/blob/1.5.x/src/master/allocator/mesos/hierarchical.cpp#L2471-L2479>
> check in the allocator (shown below) was originally introduced to
>
> help alleviate the situation where a framework receives some resources,
> but no
>
> cpu/memory, thus cannot launch a task.
>
>
> constexpr double MIN_CPUS = 0.01;constexpr Bytes MIN_MEM = Megabytes(32);
> bool HierarchicalAllocatorProcess::allocatable(
> const Resources& resources)
> {
> Option<double> cpus = resources.cpus();
> Option<Bytes> mem = resources.mem();
>
> return (cpus.isSome() && cpus.get() >= MIN_CPUS) ||
> (mem.isSome() && mem.get() >= MIN_MEM);
> }
>
>
> Issues
>
> However, there has been a couple of issues surfacing lately surrounding
> the check.
>
> -
> - - MESOS-8935 Quota limit "chopping" can lead to cpu-only and
> memory-only offers.
>
> We introduced fined-grained quota-allocation (MESOS-7099) in Mesos 1.5.
> When we
>
> allocate resources to a role, we'll "chop" the available resources of the
> agent up to the
>
> quota limit for the role. However, this has the unintended consequence of
> creating
>
> cpu-only and memory-only offers, even though there might be other agents
> with both
>
> cpu and memory resources available in the cluster.
>
>
> - MESOS-8626 The 'allocatable' check in the allocator is problematic with
> multi-role frameworks.
>
> Consider roleA reserved cpu/memory on an agent and roleB reserved disk on
> the same agent.
>
> A framework under both roleA and roleB will not be able to get the
> reserved disk due to the
>
> allocatable check. With the introduction of resource providers, the
> similar situation will
>
> become more common.
>
> Proposed change
>
> Instead of hardcoding a one-size-fits-all value in Mesos, we are proposing
> to add a new master flag
>
> min_allocatable_resources. It specifies one or more scalar resources
> quantities that define the
>
> minimum allocatable resources for the allocator. The allocator will only
> offer resources that are more
>
> than at least one of the specified resources. The default behavior *is
> backward compatible* i.e.
>
> by default, the flag is set to “cpus:0.01|mem:32”.
>
> Usage
>
> The flag takes in either a simple text of resource(s) delimited by a bar
> (|) or a JSON array of JSON
>
> formatted resources. Note, the input should be “pure” scalar quantities
> i.e. the specified resource(s)
>
> should only have name, type (set to scalar) and scalar fields set.
>
>
> Examples:
>
> - - To eliminate cpu or memory only offer due to the quota chopping,
> - we could set the flag to “cpus:0.01;mem:32”
> -
> - - To enable offering disk only offer, we could set the flag to
> “disk:32”
> -
> - - For both, we could set the flag to “cpus:0.01;mem:32|disk:32”.
> - Then the allocator will only offer resources that at least contain
> “cpus:0.01;mem:32”
> - OR resources that at least contain “disk:32”.
>
>
> Let me know what you think! Thanks!
>
>
> -Meng
>
>