You are viewing a plain text version of this content. The canonical link for it is here.
Posted to submarine-dev@hadoop.apache.org by zh...@apache.org on 2019/10/12 08:01:15 UTC
[hadoop-submarine] branch master updated: SUBMARINE-221. Update
submarine version to 0.3.0-SNAPSHOT in mini-submarine.
This is an automated email from the ASF dual-hosted git repository.
zhouquan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hadoop-submarine.git
The following commit(s) were added to refs/heads/master by this push:
new cc6b188 SUBMARINE-221. Update submarine version to 0.3.0-SNAPSHOT in mini-submarine.
cc6b188 is described below
commit cc6b1882c56ac2af3ad7ba7e604d5a9b57f04cb4
Author: Zac Zhou <yu...@gmail.com>
AuthorDate: Tue Oct 8 22:05:00 2019 +0800
SUBMARINE-221. Update submarine version to 0.3.0-SNAPSHOT in mini-submarine.
### What is this PR for?
The way to use submarine is quite different from the version 0.2.0. We need to update mini-submarine with the latest code so that users can try submarine more conveniently.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-221
### How should this be tested?
https://travis-ci.org/yuanzac/hadoop-submarine/builds/595068622
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? Yes
* Does this needs documentation? Yes
Author: Zac Zhou <yu...@gmail.com>
Closes #35 from yuanzac/topic/SUBMARINE-221 and squashes the following commits:
5b4193f [Zac Zhou] YarnServiceRuntimeFactory runtime class name is incorrect.
1401a60 [Zac Zhou] fix image version in README.
99dc118 [Zac Zhou] SUBMARINE-221. Update submarine version to 0.3.0-SNAPSHOT in mini-submarine.
---
dev-support/mini-submarine/README.md | 14 +++++++-------
dev-support/mini-submarine/build_mini-submarine.sh | 4 ++--
dev-support/mini-submarine/conf/submarine.xml | 4 ++--
.../mini-submarine/submarine/build_python_virtual_env.sh | 4 +---
.../submarine/run_customized_submarine-all_mnist.sh | 2 +-
.../mini-submarine/submarine/run_submarine_mnist_tony.sh | 11 ++++++-----
pom.xml | 5 +++++
submarine-core/pom.xml | 4 ++++
8 files changed, 28 insertions(+), 20 deletions(-)
diff --git a/dev-support/mini-submarine/README.md b/dev-support/mini-submarine/README.md
index d83a3e1..bb9560a 100644
--- a/dev-support/mini-submarine/README.md
+++ b/dev-support/mini-submarine/README.md
@@ -22,10 +22,10 @@ This is a docker image built for submarine development and quick start test.
#### Use the image we provide
-> Tag 0.2.0 indicates the version number of hadoop submarine in images
+> Tag 0.3.0-SNAPSHOT indicates the version number of hadoop submarine in images
```
-docker pull hadoopsubmarine/mini-submarine:0.2.0
+docker pull hadoopsubmarine/mini-submarine:0.3.0-SNAPSHOT
```
#### Create image by yourself
@@ -39,7 +39,7 @@ docker pull hadoopsubmarine/mini-submarine:0.2.0
#### Run mini-submarine image
```
-docker run -it -h submarine-dev --net=bridge --privileged -P local/mini-submarine:0.2.0 /bin/bash
+docker run -it -h submarine-dev --net=bridge --privileged -P local/mini-submarine:0.3.0-SNAPSHOT /bin/bash
# In the container, use root user to bootstrap hdfs and yarn
/tmp/hadoop-config/bootstrap.sh
@@ -48,7 +48,7 @@ docker run -it -h submarine-dev --net=bridge --privileged -P local/mini-submarin
yarn node -list -showDetails
```
-If you pull the image directly, please replace "local/mini-submarine:0.2.0" with "hadoopsubmarine/mini-submarine:0.2.0".
+If you pull the image directly, please replace "local/mini-submarine:0.3.0-SNAPSHOT" with "hadoopsubmarine/mini-submarine:0.3.0-SNAPSHOT".
#### You should see info like this:
@@ -155,7 +155,7 @@ When using mini-submarine, you can debug submarine client, applicationMaster and
Run the following command to start mini-submarine.
```
-docker run -it -P -h submarine-dev --net=bridge --expose=8000 --privileged local/mini-submarine:0.2.0 /bin/bash
+docker run -it -P -h submarine-dev --net=bridge --expose=8000 --privileged local/mini-submarine:0.3.0-SNAPSHOT /bin/bash
```
Debug submarine client with the parameter "--debug"
@@ -184,7 +184,7 @@ Then port 32804 can be used for remote debug.
Run the following command to start mini-submarine.
```
-docker run -it -P -h submarine-dev --net=bridge --expose=8001 --privileged local/mini-submarine:0.2.0 /bin/bash
+docker run -it -P -h submarine-dev --net=bridge --expose=8001 --privileged local/mini-submarine:0.3.0-SNAPSHOT /bin/bash
```
Add the following configuration in the file /usr/local/hadoop/etc/hadoop/tony.xml.
@@ -204,7 +204,7 @@ And the debug port mapping can be obtained using the way as [Debug submarine cli
Run the following command to start mini-submarine.
```
-docker run -it -P -h submarine-dev --net=bridge --expose=8002 --privileged local/mini-submarine:0.2.0 /bin/bash
+docker run -it -P -h submarine-dev --net=bridge --expose=8002 --privileged local/mini-submarine:0.3.0-SNAPSHOT /bin/bash
```
Add the following configuration in the file /usr/local/hadoop/etc/hadoop/tony.xml.
diff --git a/dev-support/mini-submarine/build_mini-submarine.sh b/dev-support/mini-submarine/build_mini-submarine.sh
index 198a815..f5df7c7 100755
--- a/dev-support/mini-submarine/build_mini-submarine.sh
+++ b/dev-support/mini-submarine/build_mini-submarine.sh
@@ -51,9 +51,9 @@ download_package "spark-${spark_v}-bin-hadoop2.7.tgz" "http://mirrors.tuna.tsing
download_package "zookeeper-3.4.14.tar.gz" "http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14"
cd ../..
-mysql_connector_exists=$(find -L "submarine-dist/target" -name "submarine-dist-${submarine_v}*.tar.gz")
+submarine_dist_exists=$(find -L "submarine-dist/target" -name "submarine-dist-${submarine_v}*.tar.gz")
# Build source code if the package doesn't exist.
-if [[ -z "${mysql_connector_exists}" ]]; then
+if [[ -z "${submarine_dist_exists}" ]]; then
mvn clean package -DskipTests
fi
diff --git a/dev-support/mini-submarine/conf/submarine.xml b/dev-support/mini-submarine/conf/submarine.xml
index 0ae7100..7d7fb85 100644
--- a/dev-support/mini-submarine/conf/submarine.xml
+++ b/dev-support/mini-submarine/conf/submarine.xml
@@ -19,9 +19,9 @@
<configuration>
<property>
<name>submarine.runtime.class</name>
- <value>org.apache.hadoop.yarn.submarine.runtimes.tony.TonyRuntimeFactory</value>
+ <value>org.apache.submarine.runtimes.tony.TonyRuntimeFactory</value>
<!--
- <value>org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceRuntimeFactory</value>
+ <value>org.apache.submarine.runtimes.yarnservice.YarnServiceRuntimeFactory</value>
-->
</property>
</configuration>
diff --git a/dev-support/mini-submarine/submarine/build_python_virtual_env.sh b/dev-support/mini-submarine/submarine/build_python_virtual_env.sh
index 44769f1..0bb6088 100755
--- a/dev-support/mini-submarine/submarine/build_python_virtual_env.sh
+++ b/dev-support/mini-submarine/submarine/build_python_virtual_env.sh
@@ -17,12 +17,10 @@
wget https://files.pythonhosted.org/packages/33/bc/fa0b5347139cd9564f0d44ebd2b147ac97c36b2403943dbee8a25fd74012/virtualenv-16.0.0.tar.gz
tar xf virtualenv-16.0.0.tar.gz
-wget https://github.com/linkedin/TonY/releases/download/v0.3.18/tony-cli-0.3.18-all.jar
-mv tony-cli-0.3.18-all.jar /opt/hadoop-submarine-dist-${SUBMARINE_VER}-hadoop-3.1/tony-cli-0.3.18-all.jar
# Make sure to install using Python 3, as TensorFlow only provides Python 3 artifacts
python3 virtualenv-16.0.0/virtualenv.py venv
. venv/bin/activate
-pip install tensorflow==1.13.1
+pip3 install tensorflow==1.13.1
zip -r myvenv.zip venv
deactivate
diff --git a/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh b/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
index e190b19..f2432d2 100755
--- a/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
+++ b/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
@@ -64,4 +64,4 @@ org.apache.submarine.client.cli.Cli job run --name tf-job-002 \
--worker_launch_cmd "${WORKER_CMD}" \
--ps_launch_cmd "myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 --data_dir /tmp/data --working_dir /tmp/mode" \
--insecure \
---conf tony.containers.resources=/home/yarn/submarine/myvenv.zip#archive,/home/yarn/submarine/mnist_distributed.py,/tmp/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar
+--conf tony.containers.resources=/home/yarn/submarine/myvenv.zip#archive,/home/yarn/submarine/mnist_distributed.py,/tmp/submarine-dist-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar
diff --git a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
index 88fa073..61cc3f1 100755
--- a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
+++ b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
@@ -47,10 +47,11 @@ else
WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 --data_dir /tmp/data --working_dir /tmp/mode"
fi
-SUBMARINE_VERSION=${SUBMARINE_VER}-hadoop-3.1
-SUBMARINE_HOME=/opt/hadoop-submarine-dist-${SUBMARINE_VERSION}
-CLASSPATH=$(hadoop classpath --glob):${SUBMARINE_HOME}/hadoop-submarine-core-${SUBMARINE_VER}.jar:${SUBMARINE_HOME}/hadoop-submarine-tony-runtime-${SUBMARINE_VER}.jar:${SUBMARINE_HOME}/tony-cli-0.3.18-all.jar \
-${JAVA_CMD} org.apache.hadoop.yarn.submarine.client.cli.Cli job run --name tf-job-001 \
+SUBMARINE_VERSION=0.3.0-SNAPSHOT
+HADOOP_VERSION=2.9
+
+${JAVA_CMD} -cp /opt/submarine-current/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar:/usr/local/hadoop/etc/hadoop \
+ org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
--framework tensorflow \
--verbose \
--input_path "" \
@@ -61,4 +62,4 @@ ${JAVA_CMD} org.apache.hadoop.yarn.submarine.client.cli.Cli job run --name tf-jo
--worker_launch_cmd "${WORKER_CMD}" \
--ps_launch_cmd "myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 --data_dir /tmp/data --working_dir /tmp/mode" \
--insecure \
- --conf tony.containers.resources=/home/yarn/submarine/myvenv.zip#archive,/home/yarn/submarine/mnist_distributed.py,${SUBMARINE_HOME}/tony-cli-0.3.18-all.jar
+ --conf tony.containers.resources=/home/yarn/submarine/myvenv.zip#archive,/home/yarn/submarine/mnist_distributed.py,/opt/submarine-current/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar
diff --git a/pom.xml b/pom.xml
index fcbe279..810e430 100644
--- a/pom.xml
+++ b/pom.xml
@@ -186,6 +186,11 @@
<version>${jackson-annotations.version}</version>
</dependency>
<dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>${jackson-annotations.version}</version>
+ </dependency>
+ <dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
<version>${guice.version}</version>
diff --git a/submarine-core/pom.xml b/submarine-core/pom.xml
index cd3a2f8..7595c3f 100644
--- a/submarine-core/pom.xml
+++ b/submarine-core/pom.xml
@@ -125,6 +125,10 @@
</exclusions>
</dependency>
<dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </dependency>
+ <dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>${commons-collections.version}</version>