[jira] [Commented] (YARN-7224) Support GPU isolation for docker container

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191803#comment-16191803 ] 

Wangda Tan commented on YARN-7224:
----------------------------------

YARN-6620 added support of GPU isolation in NM side, which only supports non-docker containers. We need to add support to help docker containers launched by YARN can utilize GPUs.

Currently, YARN launches docker containers inside cgroups. After YARN-6852/YARN-6620, we have finished cgroup-based GPU isolation logics. This patch is to address issues when docker container is being used.

There're several issues:
1. GPU driver and nvidia libraries: If GPU drivers and NV libraries are pre-packaged inside docker image, it could conflict to driver and nvidia-libraries installed on Host OS. An alternative solution is to detect Host OS's installed drivers and devices, mount it when launch docker container. Please refer to \[1\] for more details. 

2. Image detection: 
From \[2\], the challenge is: 
bq. Mounting user-level driver libraries and device files clobbers the environment of the container, it should be done only when the container is running a GPU application. The challenge here is to determine if a given image will be using the GPU or not. We should also prevent launching containers based on a Docker image that is incompatible with the host NVIDIA driver version, you can find more details on this wiki page.

3. GPU isolation:
We have already done this in YARN-6852/YARN-6620. 

*Proposal:*
My plan is to use nvidia-docker-plugin \[3\] to address #1, this is the same solution used by K8S \[4\]. #2 could be addressed in a separate JIRA.

We won't ship nvidia-docker-plugin with out releases and we require cluster admin to preinstall nvidia-docker-plugin to use GPU+docker support on YARN. "nvidia-docker" is a wrapper of docker binary which can address #3 as well, however "nvidia-docker" doesn't provide same semantics of docker, and it needs to setup additional environments such as PATH/LD_LIBRARY_PATH to use it. To avoid introducing additional issues, we plan to use nvidia-docker-plugin + docker binary approach.

\[1\] https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver
\[2\] https://github.com/NVIDIA/nvidia-docker/wiki/Image-inspection
\[3\] https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin
\[4\] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

> Support GPU isolation for docker container
> ------------------------------------------
>
>                 Key: YARN-7224
>                 URL: https://issues.apache.org/jira/browse/YARN-7224
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> YARN-6620 added support of GPU isolation in NM side, which only supports non-docker containers. We need to add support to help docker containers launched by YARN can utilize GPUs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org