You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by "huiyangjian (Jira)" <ji...@apache.org> on 2019/12/05 06:15:00 UTC
[jira] [Updated] (SUBMARINE-308) support PyTorch with TonY runtime
[ https://issues.apache.org/jira/browse/SUBMARINE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
huiyangjian updated SUBMARINE-308:
----------------------------------
Description:
1.Hope to support stand-alone and distributed.
2.Hopefully, pytorch case will be available on submarine github.
3.Referring to the pytorch case in the documentation, the following script was run without success.
{code:java}
//代码占位符
CLASSPATH=`hadoop classpath --glob`:/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar \
java org.apache.submarine.client.cli.Cli \
job run --name ${APP_NAME} \
--env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ \
--env DOCKER_HADOOP_HDFS_HOME=/app/hadoop-3.2.1 \
--env HADOOP_HOME=/hadoop-3.2.1 \
--env HADOOP_YARN_HOME=/hadoop-3.2.1 \
--env HADOOP_COMMON_HOME=/hadoop-3.2.1 \
--env HADOOP_HDFS_HOME=/hadoop-3.2.1 \
--env HADOOP_CONF_DIR=/hadoop-3.2.1/etc/hadoop \
--env PYTHONUNBUFFERED="0" \
--env TZ="Asia/Shanghai" \
--env YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=${LOCAL_PATH}/:/home/test \
--queue dev \
--input_path hdfs://cluster/user/work/tensorflow/data/ \
--docker_image jx-bd-hadoop13.zeus.lianjia.com:801/runonce/tf-1.13.1-pytorch-0.4-gpu-base:0.0.1 \
--num_workers 1 \
--worker_resources memory=16G,vcores=2,gpu=1 \
--worker_launch_cmd "export CLASSPATH=\$(/app/hadoop-3.2.1/bin/hadoop classpath --glob) && cd /home/test/pth && python ../pth/train_pth.py" \
--localization /home/local/test/cifar10_estimator:./submarine_algorithm
--verbose \
--conf tony.containers.resources=/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar \
--conf tony.application.framework=pytorch
{code}
was:
1.Hope to support stand-alone and distributed.
2.Hopefully, pytorch case will be available on submarine github.
> support PyTorch with TonY runtime
> ---------------------------------
>
> Key: SUBMARINE-308
> URL: https://issues.apache.org/jira/browse/SUBMARINE-308
> Project: Apache Submarine
> Issue Type: New Feature
> Components: Doc, Submarine Server
> Reporter: huiyangjian
> Priority: Major
> Fix For: 0.3.0
>
>
> 1.Hope to support stand-alone and distributed.
> 2.Hopefully, pytorch case will be available on submarine github.
> 3.Referring to the pytorch case in the documentation, the following script was run without success.
> {code:java}
> //代码占位符
> CLASSPATH=`hadoop classpath --glob`:/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar \
> java org.apache.submarine.client.cli.Cli \
> job run --name ${APP_NAME} \
> --env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ \
> --env DOCKER_HADOOP_HDFS_HOME=/app/hadoop-3.2.1 \
> --env HADOOP_HOME=/hadoop-3.2.1 \
> --env HADOOP_YARN_HOME=/hadoop-3.2.1 \
> --env HADOOP_COMMON_HOME=/hadoop-3.2.1 \
> --env HADOOP_HDFS_HOME=/hadoop-3.2.1 \
> --env HADOOP_CONF_DIR=/hadoop-3.2.1/etc/hadoop \
> --env PYTHONUNBUFFERED="0" \
> --env TZ="Asia/Shanghai" \
> --env YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=${LOCAL_PATH}/:/home/test \
> --queue dev \
> --input_path hdfs://cluster/user/work/tensorflow/data/ \
> --docker_image jx-bd-hadoop13.zeus.lianjia.com:801/runonce/tf-1.13.1-pytorch-0.4-gpu-base:0.0.1 \
> --num_workers 1 \
> --worker_resources memory=16G,vcores=2,gpu=1 \
> --worker_launch_cmd "export CLASSPATH=\$(/app/hadoop-3.2.1/bin/hadoop classpath --glob) && cd /home/test/pth && python ../pth/train_pth.py" \
> --localization /home/local/test/cifar10_estimator:./submarine_algorithm
> --verbose \
> --conf tony.containers.resources=/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar \
> --conf tony.application.framework=pytorch
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org