Question

[Gregory Stoner <an...@icloud.com.INVALID>]

Hello 

Part fo the Gang Scheduling work in 0.9, are you also looking topology awareness at rack and cluster level as well to use the scheduler with   distributed Deep Learning  training.   There are uses case where cluster can have smaller affinity zone of nodes ( CPU + Accelerators)    and lower network hop count so you want schedule smaller job in these zones  as well larger jobs  running job over full cluster ( CPU + Accelerators) 

Greg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org

Re: Question

[Weiwei Yang <ww...@apache.org>]

Hi Greg

One of the main reasons to support Gang scheduling is to support DL use
cases. Yes, we are looking at this. Do you want to share which framework
you are looking at for this?
For the topology awareness, it is not part of the scope today. I would love
to know more about the requirement before digging into the
technical details.
Are you saying the cluster has a small number of nodes that have CPU +
Accelerators and you want the scheduler to schedule smaller jobs to them as
much as possible? And do not want to attach this preference for larger jobs.

On Wed, Jun 17, 2020 at 10:26 AM Gregory Stoner
<an...@icloud.com.invalid> wrote:

> Hello
>
> Part fo the Gang Scheduling work in 0.9, are you also looking topology
> awareness at rack and cluster level as well to use the scheduler with
>  distributed Deep Learning  training.   There are uses case where cluster
> can have smaller affinity zone of nodes ( CPU + Accelerators)    and lower
> network hop count so you want schedule smaller job in these zones  as well
> larger jobs  running job over full cluster ( CPU + Accelerators)
>
> Greg
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
> For additional commands, e-mail: dev-help@yunikorn.apache.org
>
>