[GitHub] [submarine] yuanzac commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet

[GitBox <gi...@apache.org>]

yuanzac commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#discussion_r410679393
 
 

 ##########
 File path: docs/helper/TonYRuntimeGuide.md
 ##########
 @@ -187,23 +184,117 @@ You should then be able to see links and status of the jobs from command line:
 ### With Docker
 
 ```
-SUBMARINE_VERSION=0.2.0
-SUBMARINE_HOME=path-to/hadoop-submarine-dist-0.2.0-hadoop-3.1
-CLASSPATH=$(hadoop classpath --glob):${SUBMARINE_HOME}/hadoop-submarine-core-${SUBMARINE_VERSION}.jar:${SUBMARINE_HOME}/hadoop-submarine-tony-runtime-${SUBMARINE_VERSION}.jar:path-to/tony-cli-0.3.13-all.jar \
-java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
- --docker_image hadoopsubmarine/tf-1.8.0-cpu:0.0.3 \
- --input_path hdfs://pi-aw:9000/dataset/cifar-10-data \
+SUBMARINE_VERSION=0.4.0-SNAPSHOT
+SUBMARINE_HADOOP_VERSION=3.1
+CLASSPATH=$(hadoop classpath --glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar \
+java org.apache.submarine.client.cli.Cli job run --name py-job-001 \
+ --framework pytorch
+ --docker_image pytorch-latest-gpu:0.0.1 \
+ --input_path "" \
+ --num_workers 1 \
  --worker_resources memory=3G,vcores=2 \
- --worker_launch_cmd "export CLASSPATH=\$(/hadoop-3.1.0/bin/hadoop classpath --glob) && cd /test/models/tutorials/image/cifar10_estimator && python cifar10_main.py --data-dir=%input_path% --job-dir=%checkpoint_path% --train-steps=10000 --eval-batch-size=16 --train-batch-size=16 --variable-strategy=CPU --num-gpus=0 --sync" \
+ --worker_launch_cmd "cd /test/ && python cifar10_tutorial.py" \
  --env JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- --env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.0 \
+ --env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.2 \
  --env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- --env HADOOP_HOME=/hadoop-3.1.0 \
- --env HADOOP_YARN_HOME=/hadoop-3.1.0 \
- --env HADOOP_COMMON_HOME=/hadoop-3.1.0 \
- --env HADOOP_HDFS_HOME=/hadoop-3.1.0 \
- --env HADOOP_CONF_DIR=/hadoop-3.1.0/etc/hadoop \
- --conf tony.containers.resources=PATH_TO_TONY_CLI_JAR/tony-cli-0.3.2-all.jar \
- --conf tony.application.framework=pytorch
+ --env HADOOP_HOME=/hadoop-3.1.2 \
+ --env HADOOP_YARN_HOME=/hadoop-3.1.2 \
+ --env HADOOP_COMMON_HOME=/hadoop-3.1.2 \
+ --env HADOOP_HDFS_HOME=/hadoop-3.1.2 \
+ --env HADOOP_CONF_DIR=/hadoop-3.1.2/etc/hadoop \
+ --conf tony.containers.resources=path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
+```
+
+## Launch MXNet Application:
+
+### Without Docker
+
+You need:
+
+* Build a Python virtual environment with MXNet installed
+* A cluster with Hadoop 2.7 or above.
 
 Review comment:
   We've upgrade hadoop dependency to 2.9. We'd better change to Hadoop 2.9

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services