You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Chen Qingcha (JIRA)" <ji...@apache.org> on 2018/05/03 09:12:00 UTC
[jira] [Updated] (YARN-7481) Gpu locality support for Better AI
scheduling
[ https://issues.apache.org/jira/browse/YARN-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chen Qingcha updated YARN-7481:
-------------------------------
Attachment: (was: hadoop-2.7.2-gpu-port.patch)
> Gpu locality support for Better AI scheduling
> ---------------------------------------------
>
> Key: YARN-7481
> URL: https://issues.apache.org/jira/browse/YARN-7481
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, RM, yarn
> Affects Versions: 2.7.2
> Reporter: Chen Qingcha
> Priority: Major
> Fix For: 2.7.2
>
> Attachments: GPU locality support for Job scheduling.pdf, hadoop-2.7.2-gpu.patch, hadoop-2.7.2.port-gpu.patch
>
> Original Estimate: 1,344h
> Remaining Estimate: 1,344h
>
> We enhance Hadoop with GPU support for better AI job scheduling.
> Currently, YARN-3926 also supports GPU scheduling, which treats GPU as countable resource.
> However, GPU placement is also very important to deep learning job for better efficiency.
> For example, a 2-GPU job runs on gpu {0,1} could be faster than run on gpu {0, 7}, if GPU 0 and 1 are under the same PCI-E switch while 0 and 7 are not.
> We add the support to Hadoop 2.7.2 to enable GPU locality scheduling, which support fine-grained GPU placement.
> A 64-bits bitmap is added to yarn Resource, which indicates both GPU usage and locality information in a node (up to 64 GPUs per node). '1' means available and '0' otherwise in the corresponding position of the bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org