[jira] [Commented] (MESOS-5377) Improve DRF behavior with scarce resources.

["Benjamin Mahler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358190#comment-15358190 ] 

Benjamin Mahler commented on MESOS-5377:
----------------------------------------

Mitigations have been provided via a GPU framework capability (MESOS-5634) (which of course, GPU specific) and by allowing operators to exclude resources from fair sharing (see MESOS-5758).

The GPU framework capability helps to reduce the likelihood that non-GPU workloads starve out GPU workloads that want to run on the GPU machines. There are caveats to this, for example:

(1) If the framework is non-cooperative, it may fill GPU machines with non-GPU workloads, and there is currently no revocation mechanism to help evict these to make place for the GPU workloads.

(2) A mixed-workload framework (one that runs both GPU and non-GPU workloads) cannot tell in general if an offer is from an agent with GPUs present, so it must use attributes to *guarantee* that it does not place non-GPU workloads on the GPU machine.

The fairness exclusion list allows the operator to ensure that the GPU allocation does not quickly dominate the share of the role.

> Improve DRF behavior with scarce resources.
> -------------------------------------------
>
>                 Key: MESOS-5377
>                 URL: https://issues.apache.org/jira/browse/MESOS-5377
>             Project: Mesos
>          Issue Type: Epic
>          Components: allocation
>            Reporter: Benjamin Mahler
>            Assignee: Guangya Liu
>
> The allocator currently uses the notion of Weighted [Dominant Resource Fairness|https://www.cs.berkeley.edu/~alig/papers/drf.pdf] (WDRF) to establish a linear notion of fairness across allocation roles.
> DRF behaves well for resources that are present within each machine in a cluster (e.g. CPUs, memory, disk). However, some resources (e.g. GPUs) are only present on a subset of machines in the cluster.
> Consider the behavior when there are the following agents in a cluster:
> 1000 agents with (cpus:4,mem:1024,disk:1024)
> 1 agent with (gpus:1,cpus:4,mem:1024,disk:1024)
> If a role wishes to use both GPU and non-GPU resources for tasks, consuming 1 GPU will lead DRF to consider the role to have a 100% share of the cluster, since it consumes 100% of the GPUs in the cluster. This framework will then not receive any other offers.
> Among possible improvements, fairness can have understanding of resource packages. In a sense there is 1 GPU package that is competed on and 1000 non-GPU packages competed on, and ideally a role's consumption of the single GPU package does not have a large effect on the role's access to the other 1000 non-GPU packages.
> In the interim, we should consider having a recommended way to deal with scarce resources in the current model.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)