You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by "Zhankun Tang (Jira)" <ji...@apache.org> on 2019/12/05 14:03:00 UTC
[jira] [Updated] (SUBMARINE-308) Support PyTorch with TonY runtime
[ https://issues.apache.org/jira/browse/SUBMARINE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhankun Tang updated SUBMARINE-308:
-----------------------------------
Summary: Support PyTorch with TonY runtime (was: support PyTorch with TonY runtime)
> Support PyTorch with TonY runtime
> ---------------------------------
>
> Key: SUBMARINE-308
> URL: https://issues.apache.org/jira/browse/SUBMARINE-308
> Project: Apache Submarine
> Issue Type: New Feature
> Components: Doc, Submarine Server
> Reporter: huiyangjian
> Priority: Major
> Fix For: 0.3.0
>
> Attachments: localization-error.jpg, no-pytorch.jpg, yarn-mount-error.jpg
>
>
> 1.Hope to support stand-alone and distributed.
> 2.Hopefully, pytorch case will be available on submarine github.
> 3.Referring to the pytorch case in the documentation, the following script was run without success.
> {code:java}
> CLASSPATH=`hadoop classpath --glob`:/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar \
> java org.apache.submarine.client.cli.Cli \
> job run --name ${APP_NAME} \
> --env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ \
> --env DOCKER_HADOOP_HDFS_HOME=/app/hadoop-3.2.1 \
> --env HADOOP_HOME=/hadoop-3.2.1 \
> --env HADOOP_YARN_HOME=/hadoop-3.2.1 \
> --env HADOOP_COMMON_HOME=/hadoop-3.2.1 \
> --env HADOOP_HDFS_HOME=/hadoop-3.2.1 \
> --env HADOOP_CONF_DIR=/hadoop-3.2.1/etc/hadoop \
> --env PYTHONUNBUFFERED="0" \
> --env TZ="Asia/Shanghai" \
> --env YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=${LOCAL_PATH}/:/home/test \
> --queue dev \
> --input_path hdfs://cluster/user/work/tensorflow/data/ \
> --docker_image jx-bd-hadoop13.zeus.lianjia.com:801/runonce/tf-1.13.1-pytorch-0.4-gpu-base:0.0.1 \
> --num_workers 1 \
> --worker_resources memory=16G,vcores=2,gpu=1 \
> --worker_launch_cmd "export CLASSPATH=\$(/app/hadoop-3.2.1/bin/hadoop classpath --glob) && cd /home/test/pth && python ../pth/train_pth.py" \
> --localization /home/local/test/cifar10_estimator:./submarine_algorithm
> --verbose \
> --conf tony.containers.resources=/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar \
> --conf tony.application.framework=pytorch
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org