[jira] [Issue Comment Deleted] (MESOS-5377) Improve DRF behavior with scarce resources.

["Klaus Ma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MESOS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Klaus Ma updated MESOS-5377:
----------------------------
    Comment: was deleted

(was: [~bmahler], I think that's because we did not consider framework's request: {{allocator}} assumed all frameworks have the same request to all resources, the “dominant resource” is allocated resources instead of framework's request. One proposal in my mind is to implement {{requestResource}}; for your case, the dominant resources is GPU when only one task request GPU, and the dominant resources will be changed to CPU when more coming tasks request CPUs in the same framework.)

> Improve DRF behavior with scarce resources.
> -------------------------------------------
>
>                 Key: MESOS-5377
>                 URL: https://issues.apache.org/jira/browse/MESOS-5377
>             Project: Mesos
>          Issue Type: Epic
>          Components: allocation
>            Reporter: Benjamin Mahler
>
> The allocator currently uses the notion of Weighted [Dominant Resource Fairness|https://www.cs.berkeley.edu/~alig/papers/drf.pdf] (WDRF) to establish a linear notion of fairness across allocation roles.
> DRF behaves well for resources that are present within each machine in a cluster (e.g. CPUs, memory, disk). However, some resources (e.g. GPUs) are only present on a subset of machines in the cluster.
> Consider the behavior when there are the following agents in a cluster:
> 1000 agents with (cpus:4,mem:1024,disk:1024)
> 1 agent with (gpus:1,cpus:4,mem:1024,disk:1024)
> If a role wishes to use both GPU and non-GPU resources for tasks, consuming 1 GPU will lead DRF to consider the role to have a 100% share of the cluster, since it consumes 100% of the GPUs in the cluster. This framework will then not receive any other offers.
> Among possible improvements, fairness can have understanding of resource packages. In a sense there is 1 GPU package that is competed on and 1000 non-GPU packages competed on, and ideally a role's consumption of the single GPU package does not have a large effect on the role's access to the other 1000 non-GPU packages.
> In the interim, we should consider having a recommended way to deal with scarce resources in the current model.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)