You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Zhankun Tang (JIRA)" <ji...@apache.org> on 2018/08/25 05:35:00 UTC
[jira] [Commented] (YARN-8563) [Submarine] Support users to specify
Python/TF package/version/dependencies for training job.
[ https://issues.apache.org/jira/browse/YARN-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592475#comment-16592475 ]
Zhankun Tang commented on YARN-8563:
------------------------------------
[~leftnoteasy] One question:
*When and how will the prebuilt Docker images be built?*
Build-on-demand? Or we use an image hub that contains the various version combinations prebuilt in advance?
In my mind it might work like this: When user run the job, we generate the Dockerfile with the specified version of TF/Python installation sentence and build a new image based on official images.
This seems to be a more general and extensible way since the Dockerfile can also include user-specified python packages. In essence, mounting or downloading the python packages(.whl file) in docker and then pip install them seems the same way to a customized Docker build.
And it seems the "--localizations" can be also solved in build-on-demand? We download remote file/directories to local and declare it into the Dockerfile to copy to the image.
> [Submarine] Support users to specify Python/TF package/version/dependencies for training job.
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-8563
> URL: https://issues.apache.org/jira/browse/YARN-8563
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Priority: Major
>
> YARN-8561 assumes all Python / Tensorflow dependencies will be packed to docker image. In practice, user doesn't want to build docker image. Instead, user can provide python package / dependencies (like .whl), Python and TF version. And Submarine can localize specified dependencies to prebuilt base Docker images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org