You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@submarine.apache.org by GitBox <gi...@apache.org> on 2020/04/17 12:05:04 UTC
[GitHub] [submarine] lowc1012 opened a new pull request #265: SUBMARINE-370.
Add & update some documentation for MXNet
lowc1012 opened a new pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265
### What is this PR for?
Update and add documents related to MXNet.
If any other documents are required to update, please let me know.
Thanks!
### What type of PR is it?
[Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
[SUBMARINE-370](https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-370)
### How should this be tested?
[passed CI](https://travis-ci.org/github/lowc1012/submarine/builds/676124546)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] liuxunorg commented on issue #265: SUBMARINE-370. Add &
update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on issue #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#issuecomment-616009082
Will merge if no more comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] lowc1012 commented on issue #265: SUBMARINE-370. Add &
update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
lowc1012 commented on issue #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#issuecomment-615866883
@liuxunorg Thanks your suggestion. I will focus on tensorflow and pytorch in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] pingsutw commented on a change in pull request #265:
SUBMARINE-370. Add & update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#discussion_r410221431
##########
File path: docs/helper/TonYRuntimeGuide.md
##########
@@ -187,23 +184,117 @@ You should then be able to see links and status of the jobs from command line:
### With Docker
```
-SUBMARINE_VERSION=0.2.0
-SUBMARINE_HOME=path-to/hadoop-submarine-dist-0.2.0-hadoop-3.1
-CLASSPATH=$(hadoop classpath --glob):${SUBMARINE_HOME}/hadoop-submarine-core-${SUBMARINE_VERSION}.jar:${SUBMARINE_HOME}/hadoop-submarine-tony-runtime-${SUBMARINE_VERSION}.jar:path-to/tony-cli-0.3.13-all.jar \
-java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
- --docker_image hadoopsubmarine/tf-1.8.0-cpu:0.0.3 \
- --input_path hdfs://pi-aw:9000/dataset/cifar-10-data \
+SUBMARINE_VERSION=0.4.0-SNAPSHOT
+SUBMARINE_HADOOP_VERSION=3.1
+CLASSPATH=$(hadoop classpath --glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar \
+java org.apache.submarine.client.cli.Cli job run --name py-job-001 \
+ --framework pytorch
+ --docker_image pytorch-latest-gpu:0.0.1 \
+ --input_path "" \
+ --num_workers 1 \
--worker_resources memory=3G,vcores=2 \
- --worker_launch_cmd "export CLASSPATH=\$(/hadoop-3.1.0/bin/hadoop classpath --glob) && cd /test/models/tutorials/image/cifar10_estimator && python cifar10_main.py --data-dir=%input_path% --job-dir=%checkpoint_path% --train-steps=10000 --eval-batch-size=16 --train-batch-size=16 --variable-strategy=CPU --num-gpus=0 --sync" \
+ --worker_launch_cmd "cd /test/ && python cifar10_tutorial.py" \
--env JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- --env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.0 \
+ --env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.2 \
--env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- --env HADOOP_HOME=/hadoop-3.1.0 \
- --env HADOOP_YARN_HOME=/hadoop-3.1.0 \
- --env HADOOP_COMMON_HOME=/hadoop-3.1.0 \
- --env HADOOP_HDFS_HOME=/hadoop-3.1.0 \
- --env HADOOP_CONF_DIR=/hadoop-3.1.0/etc/hadoop \
- --conf tony.containers.resources=PATH_TO_TONY_CLI_JAR/tony-cli-0.3.2-all.jar \
- --conf tony.application.framework=pytorch
+ --env HADOOP_HOME=/hadoop-3.1.2 \
+ --env HADOOP_YARN_HOME=/hadoop-3.1.2 \
+ --env HADOOP_COMMON_HOME=/hadoop-3.1.2 \
+ --env HADOOP_HDFS_HOME=/hadoop-3.1.2 \
+ --env HADOOP_CONF_DIR=/hadoop-3.1.2/etc/hadoop \
+ --conf tony.containers.resources=path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
+```
+
+## Launch MXNet Application:
+
+### Without Docker
+
+You need:
+
+* Build a Python virtual environment with MXNet installed
+* A cluster with Hadoop 2.7 or above.
+
+### Building a Python virtual environment with MXNet
+
+TonY requires a Python virtual environment zip with MXNet and any needed Python libraries already installed.
+
+```
+wget https://files.pythonhosted.org/packages/33/bc/fa0b5347139cd9564f0d44ebd2b147ac97c36b2403943dbee8a25fd74012/virtualenv-16.0.0.tar.gz
+tar xf virtualenv-16.0.0.tar.gz
+
+python virtualenv-16.0.0/virtualenv.py venv
+. venv/bin/activate
+pip install mxnet==1.5.1
+zip -r myvenv.zip venv
+deactivate
```
+
+### Get the training examples
+
+Get image_classification.py from this [link](https://github.com/apache/submarine/blob/master/dev-support/mini-submarine/submarine/image_classification.py)
+
+
+```
+SUBMARINE_VERSION=0.4.0-SNAPSHOT
+SUBMARINE_HADOOP_VERSION=3.1
+CLASSPATH=$(hadoop classpath --glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar \
+java org.apache.submarine.client.cli.Cli job run --name mx-job-001 \
Review comment:
`MXNet-job-001` will be better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] yuanzac commented on a change in pull request #265:
SUBMARINE-370. Add & update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#discussion_r410679286
##########
File path: docs/helper/TonYRuntimeGuide.md
##########
@@ -110,26 +107,27 @@ java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
--env HADOOP_COMMON_HOME=/hadoop-3.1.0 \
--env HADOOP_HDFS_HOME=/hadoop-3.1.0 \
--env HADOOP_CONF_DIR=/hadoop-3.1.0/etc/hadoop \
- --conf tony.containers.resources=/home/pi/hadoop/TonY/tony-cli/build/libs/tony-cli-0.3.2-all.jar
+ --conf tony.containers.resources=path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
```
#### Notes:
1) `DOCKER_JAVA_HOME` points to JAVA_HOME inside Docker image.
2) `DOCKER_HADOOP_HDFS_HOME` points to HADOOP_HDFS_HOME inside Docker image.
-After Submarine v0.2.0, there is a uber jar `hadoop-submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar` released together with
-the `hadoop-submarine-core-${SUBMARINE_VERSION}.jar`, `hadoop-submarine-yarnservice-runtime-${SUBMARINE_VERSION}.jar` and `hadoop-submarine-tony-runtime-${SUBMARINE_VERSION}.jar`.
+We removed TonY submodule after applying [SUBMARINE-371](https://issues.apache.org/jira/browse/SUBMARINE-371) and changed to use TonY dependency directly.
+
+After Submarine v0.2.0, there is a uber jar `submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar` released together with
+the `submarine-core-${SUBMARINE_VERSION}.jar`, `submarine-yarnservice-runtime-${SUBMARINE_VERSION}.jar` and `submarine-tony-runtime-${SUBMARINE_VERSION}.jar`.
<br />
-To use the uber jar, we need to replace the three jars in previous command with one uber jar and put hadoop config directory in CLASSPATH.
-## Launch PyToch Application:
+## Launch PyTorch Application:
### Without Docker
You need:
* Build a Python virtual environment with PyTorch 0.4.0+ installed
-
+* A cluster with Hadoop 2.7 or above.
Review comment:
We've upgraded hadoop dependency to 2.9. We'd better change to Hadoop 2.9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] pingsutw commented on a change in pull request #265:
SUBMARINE-370. Add & update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#discussion_r410221867
##########
File path: docs/helper/TonYRuntimeGuide.md
##########
@@ -187,23 +184,117 @@ You should then be able to see links and status of the jobs from command line:
### With Docker
```
-SUBMARINE_VERSION=0.2.0
-SUBMARINE_HOME=path-to/hadoop-submarine-dist-0.2.0-hadoop-3.1
-CLASSPATH=$(hadoop classpath --glob):${SUBMARINE_HOME}/hadoop-submarine-core-${SUBMARINE_VERSION}.jar:${SUBMARINE_HOME}/hadoop-submarine-tony-runtime-${SUBMARINE_VERSION}.jar:path-to/tony-cli-0.3.13-all.jar \
-java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
- --docker_image hadoopsubmarine/tf-1.8.0-cpu:0.0.3 \
- --input_path hdfs://pi-aw:9000/dataset/cifar-10-data \
+SUBMARINE_VERSION=0.4.0-SNAPSHOT
+SUBMARINE_HADOOP_VERSION=3.1
+CLASSPATH=$(hadoop classpath --glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar \
+java org.apache.submarine.client.cli.Cli job run --name py-job-001 \
Review comment:
`PyTorch-job-001` will be better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] yuanzac commented on a change in pull request #265:
SUBMARINE-370. Add & update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#discussion_r410679393
##########
File path: docs/helper/TonYRuntimeGuide.md
##########
@@ -187,23 +184,117 @@ You should then be able to see links and status of the jobs from command line:
### With Docker
```
-SUBMARINE_VERSION=0.2.0
-SUBMARINE_HOME=path-to/hadoop-submarine-dist-0.2.0-hadoop-3.1
-CLASSPATH=$(hadoop classpath --glob):${SUBMARINE_HOME}/hadoop-submarine-core-${SUBMARINE_VERSION}.jar:${SUBMARINE_HOME}/hadoop-submarine-tony-runtime-${SUBMARINE_VERSION}.jar:path-to/tony-cli-0.3.13-all.jar \
-java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
- --docker_image hadoopsubmarine/tf-1.8.0-cpu:0.0.3 \
- --input_path hdfs://pi-aw:9000/dataset/cifar-10-data \
+SUBMARINE_VERSION=0.4.0-SNAPSHOT
+SUBMARINE_HADOOP_VERSION=3.1
+CLASSPATH=$(hadoop classpath --glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar \
+java org.apache.submarine.client.cli.Cli job run --name py-job-001 \
+ --framework pytorch
+ --docker_image pytorch-latest-gpu:0.0.1 \
+ --input_path "" \
+ --num_workers 1 \
--worker_resources memory=3G,vcores=2 \
- --worker_launch_cmd "export CLASSPATH=\$(/hadoop-3.1.0/bin/hadoop classpath --glob) && cd /test/models/tutorials/image/cifar10_estimator && python cifar10_main.py --data-dir=%input_path% --job-dir=%checkpoint_path% --train-steps=10000 --eval-batch-size=16 --train-batch-size=16 --variable-strategy=CPU --num-gpus=0 --sync" \
+ --worker_launch_cmd "cd /test/ && python cifar10_tutorial.py" \
--env JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- --env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.0 \
+ --env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.2 \
--env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
- --env HADOOP_HOME=/hadoop-3.1.0 \
- --env HADOOP_YARN_HOME=/hadoop-3.1.0 \
- --env HADOOP_COMMON_HOME=/hadoop-3.1.0 \
- --env HADOOP_HDFS_HOME=/hadoop-3.1.0 \
- --env HADOOP_CONF_DIR=/hadoop-3.1.0/etc/hadoop \
- --conf tony.containers.resources=PATH_TO_TONY_CLI_JAR/tony-cli-0.3.2-all.jar \
- --conf tony.application.framework=pytorch
+ --env HADOOP_HOME=/hadoop-3.1.2 \
+ --env HADOOP_YARN_HOME=/hadoop-3.1.2 \
+ --env HADOOP_COMMON_HOME=/hadoop-3.1.2 \
+ --env HADOOP_HDFS_HOME=/hadoop-3.1.2 \
+ --env HADOOP_CONF_DIR=/hadoop-3.1.2/etc/hadoop \
+ --conf tony.containers.resources=path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
+```
+
+## Launch MXNet Application:
+
+### Without Docker
+
+You need:
+
+* Build a Python virtual environment with MXNet installed
+* A cluster with Hadoop 2.7 or above.
Review comment:
We've upgrade hadoop dependency to 2.9. We'd better change to Hadoop 2.9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] lowc1012 commented on issue #265: SUBMARINE-370. Add &
update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
lowc1012 commented on issue #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#issuecomment-615842978
@pingsutw , @yuanzac
Thanks for your suggestions.
I pushed a fixed-commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] pingsutw commented on a change in pull request #265:
SUBMARINE-370. Add & update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#discussion_r410210672
##########
File path: docs/helper/docker/mxnet/base/ubuntu-18.04/Dockerfile.gpu.mx_latest
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
+
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive \
+ apt-get install -y tzdata build-essential git ninja-build ccache libopenblas-dev \
+ libopencv-dev cmake wget python3 python3-pip python3-distutils libgomp1
+
+RUN ln -s /usr/bin/python3 /usr/bin/python
+
+# CMake 3.13 or higher is required.
+RUN pip3 install --user --upgrade "cmake==3.13.3"
+
+# Build MXNet
+WORKDIR /opt
+RUN git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet
+
+WORKDIR /opt/mxnet
+RUN cp config/linux.cmake config.cmake && \
+ rm -rf build && mkdir -p build
+
+WORKDIR /opt/mxnet/build
+RUN ~/.local/bin/cmake -GNinja .. && \
+ ~/.local/bin/cmake --build .
+
+# Install MXNet for Python
+RUN cd /opt/mxnet/python && \
+ pip3 install --user -e .
Review comment:
We could use `pip install mxnet-cu102` to install the GPU version of MXNet, It will be much faster than building by ourselves.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] asfgit closed pull request #265: SUBMARINE-370. Add &
update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [submarine] pingsutw commented on issue #265: SUBMARINE-370. Add &
update some documentation for MXNet
Posted by GitBox <gi...@apache.org>.
pingsutw commented on issue #265: SUBMARINE-370. Add & update some documentation for MXNet
URL: https://github.com/apache/submarine/pull/265#issuecomment-615909587
+1, Thanks @lowc1012
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services