You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by GitBox <gi...@apache.org> on 2019/12/20 12:15:14 UTC
[GitHub] [submarine] yuanzac opened a new pull request #131: SUBMARINE-309.
Make yarn service configuraiton optional in installation guide.
yuanzac opened a new pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131
### What is this PR for?
At this time, Submarine has two runtimes for yarn clusters, yarn service and tony. So the installation guide should work for both two rumtimes. As the installation of calico and registrydns is only for yarn service, they should be marked as optional.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-309
### How should this be tested?
https://travis-ci.org/yuanzac/hadoop-submarine/builds/627688854?utm_source=github_status&utm_medium=notification
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #131:
SUBMARINE-309. Make yarn service configuraiton optional in installation
guide.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#discussion_r361240428
##########
File path: dev-support/submarine-installer/InstallationGuideChineseVersion.md
##########
@@ -614,6 +540,182 @@ Distributed-shell + GPU + cgroup
--worker_launch_cmd "python /test/cifar10_estimator/cifar10_main.py --data-dir=hdfs://${dfs_name_service}/tmp/cifar-10-data --job-dir=hdfs://${dfs_name_service}/tmp/cifar-10-jobdir --train-steps=500 --eval-batch-size=16 --train-batch-size=16 --sync --num-gpus=1"
```
+## Yarn Service Runtime (不推荐)
+
+hadoop 3.1.0 提供了 yarn native service 功能，Submarine 可以利用 yarn native service 提交分布式机器学习任务。
+但是，由于使用 yarn native service 会引入一些额外的组件，导致部署和运维服务比较困难，因而在 Submarine 0.3.0之后 Yarn Server Runtime 不再推荐使用。我们建议直接使用
+YarnRuntime，这样可以在 yarn 2.9 上提交机器学习任务。
+开启 Yarn Service Runtime，可以参照下面的方法
+
+### 安装 Etcd
+
+运行 Submarine/install.sh 脚本，就可以在指定服务器中安装 Etcd 组件和服务自启动脚本。
+
+```shell
+$ ./Submarine/install.sh
+# 通过如下命令查看 Etcd 服务状态
+systemctl status Etcd.service
+```
+
+检查 Etcd 服务状态
+
+```shell
+$ etcdctl cluster-health
+member 3adf2673436aa824 is healthy: got healthy result from http://${etcd_host_ip1}:2379
+member 85ffe9aafb7745cc is healthy: got healthy result from http://${etcd_host_ip2}:2379
+member b3d05464c356441a is healthy: got healthy result from http://${etcd_host_ip3}:2379
+cluster is healthy
+
+$ etcdctl member list
+3adf2673436aa824: name=etcdnode3 peerURLs=http://${etcd_host_ip1}:2380 clientURLs=http://${etcd_host_ip1}:2379 isLeader=false
+85ffe9aafb7745cc: name=etcdnode2 peerURLs=http://${etcd_host_ip2}:2380 clientURLs=http://${etcd_host_ip2}:2379 isLeader=false
+b3d05464c356441a: name=etcdnode1 peerURLs=http://${etcd_host_ip3}:2380 clientURLs=http://${etcd_host_ip3}:2379 isLeader=true
+```
+其中，${etcd_host_ip*} 是etcd服务器的ip
Review comment:
是etcd服务器的ip
Add a space before and after `etcd`.
`ip` need this too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #131: SUBMARINE-309. Make
yarn service configuraiton optional in installation guide.
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on issue #131: SUBMARINE-309. Make
yarn service configuraiton optional in installation guide.
Posted by GitBox <gi...@apache.org>.
yuanzac commented on issue #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#issuecomment-568716763
@liuxunorg，
Thanks for the review. Just update the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #131:
SUBMARINE-309. Make yarn service configuraiton optional in installation
guide.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#discussion_r361073699
##########
File path: dev-support/submarine-installer/InstallationGuide.md
##########
@@ -599,3 +520,185 @@ Add configurations in container-executor.cfg
root=/sys/fs/cgroup
yarn-hierarchy=/hadoop-yarn
```
+
+## Yarn Service Runtime Requirement (Deprecated)
Review comment:
Change `Deprecated` to `Not Recommend`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #131:
SUBMARINE-309. Make yarn service configuraiton optional in installation
guide.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#discussion_r361240252
##########
File path: dev-support/submarine-installer/InstallationGuideChineseVersion.md
##########
@@ -368,83 +368,16 @@ $ python >> tf.__version__
ls -l /usr/local/nvidia/lib64 | grep libcuda.so
```
-### 安装 Etcd
-
-运行 Submarine/install.sh 脚本，就可以在指定服务器中安装 Etcd 组件和服务自启动脚本。
-
-```shell
-$ ./Submarine/install.sh
-# 通过如下命令查看 Etcd 服务状态
-systemctl status Etcd.service
-```
-
-检查 Etcd 服务状态
-
-```shell
-$ etcdctl cluster-health
-member 3adf2673436aa824 is healthy: got healthy result from http://${etcd_host_ip1}:2379
-member 85ffe9aafb7745cc is healthy: got healthy result from http://${etcd_host_ip2}:2379
-member b3d05464c356441a is healthy: got healthy result from http://${etcd_host_ip3}:2379
-cluster is healthy
-
-$ etcdctl member list
-3adf2673436aa824: name=etcdnode3 peerURLs=http://${etcd_host_ip1}:2380 clientURLs=http://${etcd_host_ip1}:2379 isLeader=false
-85ffe9aafb7745cc: name=etcdnode2 peerURLs=http://${etcd_host_ip2}:2380 clientURLs=http://${etcd_host_ip2}:2379 isLeader=false
-b3d05464c356441a: name=etcdnode1 peerURLs=http://${etcd_host_ip3}:2380 clientURLs=http://${etcd_host_ip3}:2379 isLeader=true
-```
-其中，${etcd_host_ip*} 是etcd服务器的ip
-
-
-### 安装 Calico
-
-运行 Submarine/install.sh 脚本，就可以在指定服务器中安装 Calico 组件和服务自启动脚本。
-
-```
-systemctl start calico-node.service
-systemctl status calico-node.service
-```
-
-#### 检查 Calico 网络
-
-```shell
-# 执行如下命令，注意：不会显示本服务器的状态，只显示其他的服务器状态
-$ calicoctl node status
-Calico process is running.
-
-IPv4 BGP status
-+---------------+-------------------+-------+------------+-------------+
-| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
-+---------------+-------------------+-------+------------+-------------+
-| ${host_ip1} | node-to-node mesh | up | 2018-09-21 | Established |
-| ${host_ip2} | node-to-node mesh | up | 2018-09-21 | Established |
-| ${host_ip3} | node-to-node mesh | up | 2018-09-21 | Established |
-+---------------+-------------------+-------+------------+-------------+
-
-IPv6 BGP status
-No IPv6 peers found.
-```
-
-创建docker container，验证calico网络
-
-```
-docker network create --driver calico --ipam-driver calico-ipam calico-network
-docker run --net calico-network --name workload-A -tid busybox
-docker run --net calico-network --name workload-B -tid busybox
-docker exec workload-A ping workload-B
-```
-
-
## 安装 Hadoop
-### 编译 Hadoop
-
-```
-mvn package -Pdist -DskipTests -Dtar
-```
+### 安装 Hadoop
+首先，我们通过源码编译或者直接从官网[Hadoop Homepage](https://hadoop.apache.org/)下载获取hadoop包。
Review comment:
`官网[Hadoop Homepage]` Add a space before `[Hadoop Homepage]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on issue #131: SUBMARINE-309. Make
yarn service configuraiton optional in installation guide.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on issue #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#issuecomment-569037203
Will merge if no more comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on issue #131: SUBMARINE-309. Make
yarn service configuraiton optional in installation guide.
Posted by GitBox <gi...@apache.org>.
yuanzac commented on issue #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#issuecomment-568847866
@liuxunorg,
Thanks a lot. Just updated this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #131:
SUBMARINE-309. Make yarn service configuraiton optional in installation
guide.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#discussion_r361074379
##########
File path: dev-support/submarine-installer/InstallationGuideChineseVersion.md
##########
@@ -614,6 +539,182 @@ Distributed-shell + GPU + cgroup
--worker_launch_cmd "python /test/cifar10_estimator/cifar10_main.py --data-dir=hdfs://${dfs_name_service}/tmp/cifar-10-data --job-dir=hdfs://${dfs_name_service}/tmp/cifar-10-jobdir --train-steps=500 --eval-batch-size=16 --train-batch-size=16 --sync --num-gpus=1"
```
+## Yarn Service Runtime (Deprecated)
Review comment:
## Yarn Service Runtime (不推荐)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #131:
SUBMARINE-309. Make yarn service configuraiton optional in installation
guide.
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #131: SUBMARINE-309. Make yarn service configuraiton optional in installation guide.
URL: https://github.com/apache/submarine/pull/131#discussion_r361240129
##########
File path: dev-support/submarine-installer/InstallationGuideChineseVersion.md
##########
@@ -368,83 +368,16 @@ $ python >> tf.__version__
ls -l /usr/local/nvidia/lib64 | grep libcuda.so
```
-### 安装 Etcd
-
-运行 Submarine/install.sh 脚本，就可以在指定服务器中安装 Etcd 组件和服务自启动脚本。
-
-```shell
-$ ./Submarine/install.sh
-# 通过如下命令查看 Etcd 服务状态
-systemctl status Etcd.service
-```
-
-检查 Etcd 服务状态
-
-```shell
-$ etcdctl cluster-health
-member 3adf2673436aa824 is healthy: got healthy result from http://${etcd_host_ip1}:2379
-member 85ffe9aafb7745cc is healthy: got healthy result from http://${etcd_host_ip2}:2379
-member b3d05464c356441a is healthy: got healthy result from http://${etcd_host_ip3}:2379
-cluster is healthy
-
-$ etcdctl member list
-3adf2673436aa824: name=etcdnode3 peerURLs=http://${etcd_host_ip1}:2380 clientURLs=http://${etcd_host_ip1}:2379 isLeader=false
-85ffe9aafb7745cc: name=etcdnode2 peerURLs=http://${etcd_host_ip2}:2380 clientURLs=http://${etcd_host_ip2}:2379 isLeader=false
-b3d05464c356441a: name=etcdnode1 peerURLs=http://${etcd_host_ip3}:2380 clientURLs=http://${etcd_host_ip3}:2379 isLeader=true
-```
-其中，${etcd_host_ip*} 是etcd服务器的ip
-
-
-### 安装 Calico
-
-运行 Submarine/install.sh 脚本，就可以在指定服务器中安装 Calico 组件和服务自启动脚本。
-
-```
-systemctl start calico-node.service
-systemctl status calico-node.service
-```
-
-#### 检查 Calico 网络
-
-```shell
-# 执行如下命令，注意：不会显示本服务器的状态，只显示其他的服务器状态
-$ calicoctl node status
-Calico process is running.
-
-IPv4 BGP status
-+---------------+-------------------+-------+------------+-------------+
-| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
-+---------------+-------------------+-------+------------+-------------+
-| ${host_ip1} | node-to-node mesh | up | 2018-09-21 | Established |
-| ${host_ip2} | node-to-node mesh | up | 2018-09-21 | Established |
-| ${host_ip3} | node-to-node mesh | up | 2018-09-21 | Established |
-+---------------+-------------------+-------+------------+-------------+
-
-IPv6 BGP status
-No IPv6 peers found.
-```
-
-创建docker container，验证calico网络
-
-```
-docker network create --driver calico --ipam-driver calico-ipam calico-network
-docker run --net calico-network --name workload-A -tid busybox
-docker run --net calico-network --name workload-B -tid busybox
-docker exec workload-A ping workload-B
-```
-
-
## 安装 Hadoop
-### 编译 Hadoop
-
-```
-mvn package -Pdist -DskipTests -Dtar
-```
+### 安装 Hadoop
+首先，我们通过源码编译或者直接从官网[Hadoop Homepage](https://hadoop.apache.org/)下载获取hadoop包。
Review comment:
`下载获取hadoop包。` Add a space before and after `hadoop`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org