You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Zhankun Tang (JIRA)" <ji...@apache.org> on 2016/12/30 14:50:58 UTC
[jira] [Comment Edited] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15787175#comment-15787175 ]
Zhankun Tang edited comment on YARN-6043 at 12/30/16 2:50 PM:
--------------------------------------------------------------
Design doc attached. And thanks [~drankye], [~jingcheng.du], [~HuafengWang], [~mauzhang], [~zhongmuzhong]
was (Author: tangzhankun):
Proposal attached. And thanks [~drankye], [~jingcheng.du], [~HuafengWang], [~mauzhang], [~zhongmuzhong]
> [HDL] Tensorflow on YARN
> ------------------------
>
> Key: YARN-6043
> URL: https://issues.apache.org/jira/browse/YARN-6043
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Kai Zheng
> Attachments: TensorFlow_on_YARN_proposal.pdf
>
>
> As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep Learning on Hadoop. As a beginning, we implemented a prototype running Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It allocates and launches a Tensorflow cluster in YARN dynamically, executing the job, and then destroys the cluster after the work is done. It doesn't require Python and Tensorflow binary installations be done previously on YARN nodes (on client host, Python is required if the job is written in Python). It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle (the TF core library and our JNI wrapper) and yarn-tf tool.
> In this jira we'll post our design documenting how we did it and the general approach. The prototype work is under polishing and will be public here soon.
> Filing this as unassigned as it's a team work. Your thoughts and feedback are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org